Scrapers: Difference between revisions
No edit summary |
No edit summary |
||
(29 intermediate revisions by 12 users not shown) | |||
Line 1: | Line 1: | ||
{{ | {{mininav|[[Development]] {{l2| [[Music library]] }} {{l2|[[Video library]]}} }} | ||
= Using Scrapers = | |||
Scrapers are used to contact online information providers to download metadata for video and music items that are to be added to the library. | |||
A number of scrapers are installed by default when installing {{kodi}}. Other scrapers are available for installation from the official repository. The tables at the bottom of this page list all official scrapers and their compatability. | |||
== | {| style="width: 100%;" | ||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[HOW-TO:Create_Video_Library|Creating your Video Library]]}}''' | |||
Setting up your video library for the first time? Follow this How To guide | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[HOW-TO:Create_Music_Library|Creating your Music Library]]}}''' | |||
Setting up your music library for the first time? Follow this How To guide | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Incorrect_and_missing_videos|Video scraping problems]]}}''' | |||
Something not working as expected? Check this page for common solutions. | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | | |||
| style="width: 44%; vertical-align:top;" | | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Changing Scrapers]]}}''' | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[:Category:Scraper|List of all Scraper pages]]}}''' | |||
|} | |||
= | = Scraper Development = | ||
Historically, {{kodi}} scrapers were XML based. While XML scrapers have been used successfully for well over a decade, their use has been deprecated, since the release of {{kodi}} v19 Matrix, in favour of Python based scrapers. | |||
XML scrapers have become difficult to develop and maintain as they are no longer suited to the advanced REST API's used by the information providers, which results in frequent maintenance and updates to cope with API changes. | |||
The following pages are available to help with the development process of scrapers. Although XML scrapers are deprecated, and no future XML scrapers will be accepted into the official repository, the two pages dedicated to XML scrapers contain useful information. | |||
{| style="width: 100%;" | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[Python_movie_scraper_development|Developing a Movie scraper in Python]]}}''' | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Scraper development XML|XML Scraper Development]]}}''' | |||
XML scrapers are deprecated. This page may still contain useful development information. | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[Python_tv_scraper_development|Developing a TV Show scraper in Python]]}}''' | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[HOW-TO:Write_media_scrapers|Writing an XML scraper]]}}''' | |||
XML scrapers are deprecated. This page may still contain useful development information. | |||
|} | |||
= Official Scrapers = | |||
{| | |||
! style="background-color:salmon;" | Scrapers compatible with v19 Matrix | |||
! style="background-color:#a0e75a;" | Scrapers compatible with v20 Nexus | |||
! style="background-color:skyBlue;" | Scrapers compatible with v21 Omega | |||
|- | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Matrix_add-on_repository | |||
category = Album_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Nexus_add-on_repository | |||
category = Album_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Omega_add-on_repository | |||
category = Album_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
|- | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Matrix_add-on_repository | |||
category = Artist_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Nexus_add-on_repository | |||
category = Artist_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Omega_add-on_repository | |||
category = Artist_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
|- | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Matrix_add-on_repository | |||
category = Music_video_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Nexus_add-on_repository | |||
category = Music_video_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Omega_add-on_repository | |||
category = Music_video_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
|- | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Matrix_add-on_repository | |||
category = Movie_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Nexus_add-on_repository | |||
category = Movie_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Omega_add-on_repository | |||
category = Movie_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
|- | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Matrix_add-on_repository | |||
category = TV_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Nexus_add-on_repository | |||
category = TV_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
| | |||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | |||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers | |||
|- | |||
| | |||
<dpl> | |||
category = Omega_add-on_repository | |||
category = TV_scraper_add-ons | |||
notcategory = Broken add-ons | |||
</dpl> | |||
|} | |||
|} | |||
{{updated|20}} | |||
[[Category:Karellen]] | |||
[[Category:Scraper]] | |||
[[Category:Music library]] | |||
[[Category:Video library]] | |||
[[Category:Disambiguation_pages]] | |||
[[Category | |||
[[ | |||
[[ | |||
[[ |
Latest revision as of 01:33, 24 January 2023
Development Music library Video library |
Scrapers |
Using Scrapers
Scrapers are used to contact online information providers to download metadata for video and music items that are to be added to the library.
A number of scrapers are installed by default when installing Kodi. Other scrapers are available for installation from the official repository. The tables at the bottom of this page list all official scrapers and their compatability.
Creating your Video Library
Setting up your video library for the first time? Follow this How To guide |
Creating your Music Library
Setting up your music library for the first time? Follow this How To guide | |||
Video scraping problems
Something not working as expected? Check this page for common solutions. |
||||
Changing Scrapers | List of all Scraper pages |
Scraper Development
Historically, Kodi scrapers were XML based. While XML scrapers have been used successfully for well over a decade, their use has been deprecated, since the release of Kodi v19 Matrix, in favour of Python based scrapers.
XML scrapers have become difficult to develop and maintain as they are no longer suited to the advanced REST API's used by the information providers, which results in frequent maintenance and updates to cope with API changes.
The following pages are available to help with the development process of scrapers. Although XML scrapers are deprecated, and no future XML scrapers will be accepted into the official repository, the two pages dedicated to XML scrapers contain useful information.
Developing a Movie scraper in Python | XML Scraper Development
XML scrapers are deprecated. This page may still contain useful development information. | |||
Developing a TV Show scraper in Python | Writing an XML scraper
XML scrapers are deprecated. This page may still contain useful development information. |
Official Scrapers
Scrapers compatible with v19 Matrix | Scrapers compatible with v20 Nexus | Scrapers compatible with v21 Omega | ||||||
---|---|---|---|---|---|---|---|---|
|
|
| ||||||
|
|
| ||||||
|
|
| ||||||
|
|
| ||||||
|
|
|