Scrapers - Revision history

Karellen at 01:33, 24 January 2023

2023-01-24T01:33:38Z

← Older revision		Revision as of 01:33, 24 January 2023
Line 76:		Line 76:
	category = Matrix_add-on_repository		category = Matrix_add-on_repository
	category = Album_scraper_add-ons		category = Album_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 87:		Line 87:
	category = Nexus_add-on_repository		category = Nexus_add-on_repository
	category = Album_scraper_add-ons		category = Album_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 98:		Line 98:
	category = Omega_add-on_repository		category = Omega_add-on_repository
	category = Album_scraper_add-ons		category = Album_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 110:		Line 110:
	category = Matrix_add-on_repository		category = Matrix_add-on_repository
	category = Artist_scraper_add-ons		category = Artist_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 121:		Line 121:
	category = Nexus_add-on_repository		category = Nexus_add-on_repository
	category = Artist_scraper_add-ons		category = Artist_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 132:		Line 132:
	category = Omega_add-on_repository		category = Omega_add-on_repository
	category = Artist_scraper_add-ons		category = Artist_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 144:		Line 144:
	category = Matrix_add-on_repository		category = Matrix_add-on_repository
	category = Music_video_scraper_add-ons		category = Music_video_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 155:		Line 155:
	category = Nexus_add-on_repository		category = Nexus_add-on_repository
	category = Music_video_scraper_add-ons		category = Music_video_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 166:		Line 166:
	category = Omega_add-on_repository		category = Omega_add-on_repository
	category = Music_video_scraper_add-ons		category = Music_video_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 178:		Line 178:
	category = Matrix_add-on_repository		category = Matrix_add-on_repository
	category = Movie_scraper_add-ons		category = Movie_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 189:		Line 189:
	category = Nexus_add-on_repository		category = Nexus_add-on_repository
	category = Movie_scraper_add-ons		category = Movie_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 200:		Line 200:
	category = Omega_add-on_repository		category = Omega_add-on_repository
	category = Movie_scraper_add-ons		category = Movie_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 212:		Line 212:
	category = Matrix_add-on_repository		category = Matrix_add-on_repository
	category = TV_scraper_add-ons		category = TV_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 223:		Line 223:
	category = Nexus_add-on_repository		category = Nexus_add-on_repository
	category = TV_scraper_add-ons		category = TV_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}
Line 234:		Line 234:
	category = Omega_add-on_repository		category = Omega_add-on_repository
	category = TV_scraper_add-ons		category = TV_scraper_add-ons
	notcategory = ~~Outdated~~		notcategory = Broken add-ons
	</dpl>		</dpl>
	\|}		\|}

Karellen: Updated for v20

2023-01-22T23:45:17Z

Updated for v20

Show changes

Karellen: Convert to Portal

2022-05-29T04:44:20Z

Convert to Portal

Show changes

Karellen: Remove links

2022-05-28T22:24:10Z

Remove links

← Older revision		Revision as of 22:24, 28 May 2022
Line 18:		Line 18:

	== Prerequisites ==		== Prerequisites ==
	* Knowledge of ~~[[RegEx_tutorial\|~~regular expressions]]		* Knowledge of regular expressions
	* Basic [http://en.wikipedia.org/wiki/XML XML] syntax knowledge.		* Basic [http://en.wikipedia.org/wiki/XML XML] syntax knowledge.

SistemaRayoXP: Added syntax highlight

2021-09-15T14:12:43Z

Added syntax highlight

← Older revision		Revision as of 14:12, 15 September 2021
Line 24:		Line 24:
	To see a full scraper, see the [https://github.com/xbmc/repo-scrapers/tree/gotham/metadata.themoviedb.org themoviedb reference implementation] in GIT.		To see a full scraper, see the [https://github.com/xbmc/repo-scrapers/tree/gotham/metadata.themoviedb.org themoviedb reference implementation] in GIT.
	The general layout of scraper.xml is as follows:		The general layout of scraper.xml is as follows:
			<syntaxhighlight lang="xml" enclose="div">
	<scraper>		<scraper>
	<NfoUrl>		<NfoUrl>
Line 52:		Line 52:
	</CustomFunction>		</CustomFunction>
	</scraper>		</scraper>
			</syntaxhighlight>

	If RegExp tags are being nested they are being worked through in a [http://en.wikipedia.org/wiki/Lifo lifo] manner.		If RegExp tags are being nested they are being worked through in a [http://en.wikipedia.org/wiki/Lifo lifo] manner.
Line 70:		Line 71:

	For example, the following would be wrong:		For example, the following would be wrong:
			<source lang="xml">
	<CreateSearchUrl dest="3">		<CreateSearchUrl dest="3">
	<RegExp input="$$1" output="http://www.example.com/search.php?title=\1&new=true" dest="3">		<RegExp input="$$1" output="http://www.example.com/search.php?title=\1&new=true" dest="3">
Line 76:		Line 77:
	</RegExp>		</RegExp>
	</CreateSearchUrl>		</CreateSearchUrl>
			</source>
	Use instead:		Use instead:
			<source lang="xml">
	<CreateSearchUrl dest="3">		<CreateSearchUrl dest="3">
	<RegExp input="$$1" output="http://www.example.com/search.php?title=\1&amp;new=true" dest="3">		<RegExp input="$$1" output="http://www.example.com/search.php?title=\1&amp;new=true" dest="3">
Line 84:		Line 85:
	</RegExp>		</RegExp>
	</CreateSearchUrl>		</CreateSearchUrl>
			</source>
	If you open the scraper XML file in a web browser the actual characters are shown for the entities, be careful if you cut&paste.		If you open the scraper XML file in a web browser the actual characters are shown for the entities, be careful if you cut&paste.

Line 120:		Line 121:
	=== <RegExp><expression></RegExp> ===		=== <RegExp><expression></RegExp> ===
	A RegExp is a text processing command that takes text in buffer 'input', runs <expression> on it... formats the matches with 'output' and places the result in buffer 'dest'		A RegExp is a text processing command that takes text in buffer 'input', runs <expression> on it... formats the matches with 'output' and places the result in buffer 'dest'
			<source lang="xml">
	<RegExp conditional="override" input="$$5" output="<details>\1</details>" dest="3">		<RegExp conditional="override" input="$$5" output="<details>\1</details>" dest="3">
	<expression repeat="yes" noclean="1" trim="1" clear="no">([a-zA-Z][^,]*)</expression>		<expression repeat="yes" noclean="1" trim="1" clear="no">([a-zA-Z][^,]*)</expression>
	</RegExp>		</RegExp>
			</source>
	==== <expression> ====		==== <expression> ====
	The <expression> tag holds the regular expression. Here it is: <xml>([a-zA-Z][^,])whatever (.[^-])</xml>		The <expression> tag holds the regular expression. Here it is: <xml>([a-zA-Z][^,])whatever (.[^-])</xml>
Line 148:		Line 149:

	=== <CreateSearchUrl> ===		=== <CreateSearchUrl> ===
			<source lang="xml">
	<CreateSearchUrl dest="3">		<CreateSearchUrl dest="3">
	<RegExp input="$$1" output="http://akas.imdb.com/find?s=tt;q=\1" dest="3">		<RegExp input="$$1" output="http://akas.imdb.com/find?s=tt;q=\1" dest="3">
Line 154:		Line 155:
	</RegExp>		</RegExp>
	</CreateSearchUrl>		</CreateSearchUrl>
			</source>
	'''Inputs''':<br />		'''Inputs''':<br />
	* $$1: Variable 1 holds the search string. This is usually the filename stripped by some words e.g. DVDRip, Xvid ..		* $$1: Variable 1 holds the search string. This is usually the filename stripped by some words e.g. DVDRip, Xvid ..
Line 165:		Line 166:

	=== <GetSearchResults> ===		=== <GetSearchResults> ===
			<source lang="xml">
	<GetSearchResults dest="8">		<GetSearchResults dest="8">
	<RegExp input="$$5" output="<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?><results>\1</results>" dest="8">		<RegExp input="$$5" output="<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?><results>\1</results>" dest="8">
Line 174:		Line 175:
	</RegExp>		</RegExp>
	</GetSearchResults>		</GetSearchResults>
			</source>
	'''Inputs''':<br />		'''Inputs''':<br />
	* $$1: Variable 1 hold the content of the search URL returned to CreateSearchUrl.		* $$1: Variable 1 hold the content of the search URL returned to CreateSearchUrl.

	It is the task of this function to return a list of search results in the following format:		It is the task of this function to return a list of search results in the following format:
			<syntaxhighlight lang="xml" enclose="div">
	<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>		<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
	<results>		<results>
Line 193:		Line 194:
	</entity>		</entity>
	</results>		</results>
			</syntaxhighlight>
	There can be up to 9 <url> tags within each movie tag. The webpages specified in the url tags should hold detailed information about the movie/tv series/mvid. In most cases one url will be enough. The url tag can have the argument function="nameoffunction" e.g.		There can be up to 9 <url> tags within each movie tag. The webpages specified in the url tags should hold detailed information about the movie/tv series/mvid. In most cases one url will be enough. The url tag can have the argument function="nameoffunction" e.g.
	<xml><url function="CustomFunction">theurl</url></xml>		<source lang="xml"><xml><url function="CustomFunction">theurl</url></xml></source>
	In this case other then GetDetails, "CustomFunction" will be executed after GetDetails has run.		In this case other then GetDetails, "CustomFunction" will be executed after GetDetails has run.

Line 204:		Line 205:
	=== <GetDetails> ===		=== <GetDetails> ===
	Here the actual data which is being saved in the database is being retrieved. For movies the following format should be returned:		Here the actual data which is being saved in the database is being retrieved. For movies the following format should be returned:
			<source lang="xml">
	<details>		<details>
	<title></title>		<title></title>
Line 225:		Line 226:
	<plot></plot>		<plot></plot>
	</details>		</details>
			</source>
	'''Inputs:'''<br />		'''Inputs:'''<br />
	* $$x where x= 1 to 9: Variable x holds the content of the URL specified in the <url> tag number x in GetSearchResults.		* $$x where x= 1 to 9: Variable x holds the content of the URL specified in the <url> tag number x in GetSearchResults.
Line 247:		Line 248:
	* $$1: Variable 1 holds the content of the URL specified in the url tag with the function="CustomFunction" argument.		* $$1: Variable 1 holds the content of the URL specified in the url tag with the function="CustomFunction" argument.
	e.g. for		e.g. for
			<source lang="xml">
	<url function="CustomFunction">http://www.imdb.com/title/tt0452624/</url>		<url function="CustomFunction">http://www.imdb.com/title/tt0452624/</url>
			</source>
	variable 1 would hold the content of "http://www.imdb.com/title/tt0452624/".		variable 1 would hold the content of "http://www.imdb.com/title/tt0452624/".

Karellen at 03:22, 12 July 2020

2020-07-12T03:22:27Z

← Older revision		Revision as of 03:22, 12 July 2020
Line 5:		Line 5:

	= Introduction to Scrapers =		= Introduction to Scrapers =
	Before version 2.0 of Kodi Media Center the only available web scraper we had was a ~~[[IMDB\|~~IMDb]] importer. In later versions it has become possible to create custom scrapers to collect data from almost any web-page which list information in a consistent way, and today Kodi comes with many scrapers for several languages and different websites.		Before version 2.0 of Kodi Media Center the only available web scraper we had was a IMDb importer. In later versions it has become possible to create custom scrapers to collect data from almost any web-page which list information in a consistent way, and today Kodi comes with many scrapers for several languages and different websites.

	To learn more about actually using scrapers in Kodi, please look at:		To learn more about actually using scrapers in Kodi, please look at:
	* [[Adding_video_sources~~#Scraper_Settings~~]]		* [[Adding_video_sources]]
	And to learn more about '''creating''' scrapers, please look at this article:		And to learn more about '''creating''' scrapers, please look at this article:
	* [[HOW-TO Write Media Info Scrapers]]		* [[HOW-TO:Write_media_scrapers\|HOW-TO Write Media Info Scrapers]]

	Kodi come with several scrapers for Movies, TV shows and Music Videos which are stored in xbmc\system\scrapers\video. They are just specially formatted [http://en.wikipedia.org/wiki/XML XML] files.		Kodi come with several scrapers for Movies, TV shows and Music Videos which are stored in xbmc\system\scrapers\video. They are just specially formatted [http://en.wikipedia.org/wiki/XML XML] files.
Line 18:		Line 18:

	== Prerequisites ==		== Prerequisites ==
	* Knowledge of [[~~Regular Expression (RegEx) Tutorial~~\|regular expressions]]		* Knowledge of [[RegEx_tutorial\|regular expressions]]
	* Basic [http://en.wikipedia.org/wiki/XML XML] syntax knowledge.		* Basic [http://en.wikipedia.org/wiki/XML XML] syntax knowledge.

Line 252:		Line 252:
	variable 1 would hold the content of "http://www.imdb.com/title/tt0452624/".		variable 1 would hold the content of "http://www.imdb.com/title/tt0452624/".

	~~[[Category:Manual]]~~
	[[Category:Development]]		[[Category:Development]]
	[[Category:Scraper]]		[[Category:Scraper]]
	[[Category:Video library]]		[[Category:Video library]]
	[[Category:Music library]]		[[Category:Music library]]

Karellen at 19:59, 28 February 2020

2020-02-28T19:59:21Z

← Older revision		Revision as of 19:59, 28 February 2020
Line 5:		Line 5:

	= Introduction to Scrapers =		= Introduction to Scrapers =
	Before version 2.0 of Kodi Media Center the only available web scraper we had was a [[~~IMDb (Video Library)~~\|IMDb]] importer. In later versions it has become possible to create custom scrapers to collect data from almost any web-page which list information in a consistent way, and today Kodi comes with many scrapers for several languages and different websites.		Before version 2.0 of Kodi Media Center the only available web scraper we had was a [[IMDB\|IMDb]] importer. In later versions it has become possible to create custom scrapers to collect data from almost any web-page which list information in a consistent way, and today Kodi comes with many scrapers for several languages and different websites.

	To learn more about actually using scrapers in Kodi, please look at:		To learn more about actually using scrapers in Kodi, please look at:

Davu at 08:44, 26 February 2018

2018-02-26T08:44:16Z

← Older revision		Revision as of 08:44, 26 February 2018
Line 8:		Line 8:

	To learn more about actually using scrapers in Kodi, please look at:		To learn more about actually using scrapers in Kodi, please look at:
	* [[~~Set Content (Video Library)~~]]		* [[Adding_video_sources#Scraper_Settings]]
	And to learn more about '''creating''' scrapers, please look at this article:		And to learn more about '''creating''' scrapers, please look at this article:
	* [[HOW-TO Write Media Info Scrapers]]		* [[HOW-TO Write Media Info Scrapers]]

Karellen at 07:01, 20 August 2017

2017-08-20T07:01:47Z

← Older revision		Revision as of 07:01, 20 August 2017
Line 2:		Line 2:
	{{see also\|Video library\|Music library}}		{{see also\|Video library\|Music library}}
	{{cleanup}}		{{cleanup}}
	<section begin="intro" />Scrapers (also outside the Kodi community more commonly referred to as "''[http://en.wikipedia.org/wiki/Web_scraper web scrapers]''" or "''[http://en.wikipedia.org/wiki/Importer_(computing) importers]''") are used by Kodi to go online to automatically retrieve [http://en.wikipedia.org/wiki/Metadata metadata] information and ~~artwork (~~[[~~thumbnails]]poster/cover/[[fanart~~]]) from websites for Music, Videos, Movies and TV Shows. This meta data information and artwork is then used for ~~the~~ Kodi music and video [[Library\|libraries]].<section end="intro" /><br />		<section begin="intro" />Scrapers (also outside the Kodi community more commonly referred to as "''[http://en.wikipedia.org/wiki/Web_scraper web scrapers]''" or "''[http://en.wikipedia.org/wiki/Importer_(computing) importers]''") are used by Kodi to go online to automatically retrieve [http://en.wikipedia.org/wiki/Metadata metadata] information and [[artwork]] from websites for Music, Videos, Movies and TV Shows. This meta data information and artwork is then used for Kodi's music and video [[Library\|libraries]].<section end="intro" /><br />

	= Introduction to Scrapers =		= Introduction to Scrapers =

Karellen: Add intro transclusion

2017-08-20T06:59:07Z

Add intro transclusion

← Older revision		Revision as of 06:59, 20 August 2017
Line 2:		Line 2:
	{{see also\|Video library\|Music library}}		{{see also\|Video library\|Music library}}
	{{cleanup}}		{{cleanup}}
	Scrapers (also outside the Kodi community more commonly referred to as "''[http://en.wikipedia.org/wiki/Web_scraper web scrapers]''" or "''[http://en.wikipedia.org/wiki/Importer_(computing) importers]''") are used by ~~XBMC Media Center~~ to go online to automatically retrieve [http://en.wikipedia.org/wiki/Metadata metadata] information and artwork ([[thumbnails]]poster/cover/[[fanart]]) from websites for ~~Games,~~ Music, Videos, Movies, and TV Shows. This meta data information and artwork is then used for the Kodi ~~games,~~ music, and video [[Library\|libraries]].<br />		<section begin="intro" />Scrapers (also outside the Kodi community more commonly referred to as "''[http://en.wikipedia.org/wiki/Web_scraper web scrapers]''" or "''[http://en.wikipedia.org/wiki/Importer_(computing) importers]''") are used by Kodi to go online to automatically retrieve [http://en.wikipedia.org/wiki/Metadata metadata] information and artwork ([[thumbnails]]poster/cover/[[fanart]]) from websites for Music, Videos, Movies and TV Shows. This meta data information and artwork is then used for the Kodi music and video [[Library\|libraries]].<section end="intro" /><br />

	= Introduction to Scrapers =		= Introduction to Scrapers =