Discovering data catalogs
Three different jobs:
| Job | What you want | Where to go |
|---|---|---|
| Find catalogs already in this registry | Filter by country, type, software, or URL | query-examples.md, agents/query.md |
| Find catalogs not yet registered | New portals, geoportals, repositories | This page, then CONTRIBUTING.md |
| List datasets inside a registered catalog | Remote catalog APIs, with type filters | harvest.md |
This page is for the second job: locating real catalog installations in the wild, checking they are not duplicates, and preparing them for a pull request. Coding agents should follow the shorter checklist in agents/discover.md. What to hunt next, and which prompts actually produced catalogs: agents/improve.md.
The registry records catalogs (portals, geoportals, repositories, and similar infrastructure). It does not store the datasets inside those catalogs. To crawl those datasets, see harvest.md.
Guides
| Guide | Use when |
|---|---|
| Search engines and internet maps | Google, Censys, Shodan, FOFA, URLScan, crt.sh, and similar tools |
| Agents, Cursor, ChatGPT | Configure MCP, APIs, Custom GPTs, and LLM clients to use those tools |
| Open data portals | CKAN, DKAN, EKAN, OpenDataSoft, Socrata, uData, Magda, JKAN, Junar, EntryScape, ArcGIS Hub, Idra, Liferay, POMOSAM, oPortal, OGD India, data eye, Piveau, Our Open Data, DataPress, ResourceContracts, RDF Online Repository, Guangxi, ODWeb, OpenGov |
| Geoportals | Overview; SDI stacks: discovery-geoportals-sdi.md; viewers: discovery-geoportals-viewers.md (GISApp, SmartMap, SmartGIS, ISY Map, Avinet, MAP+, EnviMAP, PISO, GDi Visios, MapGuide, Zeljko GIS, SeaSketch, XY Maps, Hajk, Origo, myCarta, AddSpatial, KortInfo, …) |
| Scientific repositories | Institutional IRs and CRIS (DSpace, Hyrax, Figshare, Pure, Converis, Omega-PSIR, Archipelago, Redivis, DABAR, OpenScience.si, …); domain repos: discovery-scientific-domain.md |
| Metadata catalogs | FAIR Data Point, Aristotle MDR, Fusion Registry, Metadata Browser, CBD Clearing-House, WMO OSCAR |
| Indicators and microdata | PxWeb, PxStat, DGBAS Web, OpenSDG, Goal Tracker, IMF NSDP, .Stat Suite, Knoema, SDMX-RI, GENESIS-Online, IBIS-PH, DHIS2, FENIX, TabNet, SparkMap, eDatos, Cancer-Rates.info, Conduent HCI, Virtual LMI, DataWarehousePro, Beyond 20/20, NADA, NESSTAR, REDATAM, Colectica, OBiBa Mica, IPUMS, StatPlanet |
| Search, ML, API, marketplaces | Data search engines (Idra, OpenAIRE), ML catalogs, API directories, data marketplaces |
Before you search
- Confirm the candidate is a catalog: it lists or serves datasets, maps, indicators, or metadata — not a news site, a single spreadsheet, or a login-only intranet.
- Search the registry first. Duplicate
linkvalues fail quality checks (DUPLICATE_LINK/DUPLICATE_LINK_NORMALIZED). - Prefer
data/scheduled/for unverified finds. Promote later; see scheduled.md.
Duplicate check against the DuckDB export:
SELECT id, name, link, catalog_type, status
FROM catalogs
WHERE lower(link) LIKE '%example.gov%'
OR id = 'examplegov';
Also search data/scheduled/ if that directory is not empty. Do not walk every YAML file under data/entities/ unless you are editing a specific record.
Where catalogs usually live
- National open-data sites:
/data,/opendata,/datasets,data.gov.*,datos.*,donnees.* - Statistics offices, mapping agencies, environmental agencies, and universities
- Local government: city/region sites,
opendata.subdomains, ArcGIS Hub sites - Research: institutional repositories, Dataverse, DSpace, GBIF IPT, re3data-listed repos
Search with the local language (datos abiertos, données ouvertes, offene daten, dados abertos, 开放数据) plus the country or city name. Restrict with site:.gov, site:.gob.*, or the national government TLD. Operators, Censys/Shodan/FOFA queries, and other indexes: discovery-search-tools.md (FOFA is the Censys alternative when Platform search is unavailable). Per-platform queries: the guides above.
Existing lists (start here)
Many platforms publish installation galleries. Cross-check each URL against the registry before adding it. High-yield sources:
| Source | Typical catalogs |
|---|---|
| CKAN ecosystem | CKAN open-data portals (automated: scripts/sync_ckan_ecosystem.py) |
| CKAN Ecosystem Insights | Weekly-crawl dashboard of every tracked CKAN portal (dataset counts, versions, reachability). Machine-readable rollup: dashboard-data/dashboard.json — diff its instances list against exports to spot unregistered or moved CKAN sites |
| Datashades | CKAN and other portals |
| data.europa.eu catalogues | EU and member-state catalogs |
| GeoNetwork gallery | GeoNetwork geoportals |
| INSPIRE geoportal | European spatial catalogs |
| re3data | Research data repositories |
| Dataverse installations | Dataverse |
| STAC Index | STAC catalogs |
| Open Data Inception | Open-data portals |
| OpenSDG community | SDG indicator sites |
| StatSilk gallery | StatPlanet indicator maps (many Flash-era; confirm live Cloud/HTML5) |
| ISTAT StatKit implementations | Istat Data Browser / StatKit hubs |
| GBIF IPT | Biodiversity IPT nodes |
| ArcGIS Hub | ArcGIS Hub sites |
| EntryScape customers | EntryScape catalogs |
| FAIR Data Point index | FAIR Data Point metadata catalogs |
| KOBV OPUS 4 references | OPUS institutional repositories |
| About RADAR | RADAR Cloud and RADAR Local research data repositories |
| DHIS2 | National HMIS / public health indicator portals |
| Yoda | Dutch university research-data vaults |
| OpenAIRE CONNECT | National and community research gateways |
| OpenAIRE Graph data sources | Research data repositories harvested by OpenAIRE (automated: scripts/extract_openaire_portals.py) |
| IPUMS | Harmonized census and survey microdata collections |
| Symbiota portals | Biodiversity collection CMS portals |
| openEO | EO cloud-processing backends |
| Breedbase | Crop breeding information systems |
| Landfolio portals | Trimble mining and land cadastre map portals |
| G3W-SUITE | Italian QGIS WebGIS |
| Hajk users | Swedish Hajk webGIS |
| Origo / Origosamverkan | Swedish Origo municipal webbkartor |
| Aveki myCarta | Swedish myCarta WebMap |
| Icebound AddSpatial | Swedish AddSpatial municipal webbkartor |
| ResourceContracts | Oil/gas/mining contract repositories |
| RDF Online Repository | Mining and petroleum license-transparency portals |
| FAOSTAT / FENIX | FAO indicator catalogs (AMIS, DAD-IS, WIEWS, GIFT) |
| DATASUS TabNet | Brazilian health tabulators |
| SparkMap | US community and health indicator hubs |
| Sentinel Hub Catalog | EO STAC collections |
| IMF DSBB | National Summary Data Pages (e-GDDS / SDDS / SDDS Plus) |
| InterMine registry | Organism and project mines |
| GRIN-Global | National / centre genebank installs |
| cBioPortal | Public and hospital cancer-genomics instances |
| TalkBank | Spoken-language transcript banks (CHILDES, AphasiaBank, FluencyBank) |
| NOMAD | Materials archive and self-hosted Oasis |
| XNAT | Neuroimaging archives |
| Open Microscopy / IDR | OMERO public image archives |
| ROAR | Open-access repositories |
| OpenDOAR | Directory of open-access repositories (country + software facets) |
| CoreTrustSeal | Certified repositories — keep only those with a public dataset catalog |
| ODIS catalogue | Ocean Data and Information System catalog records |
| WMO WIS2 GDC | WIS2 Global Discovery Catalogue (node catalogs, not MQTT topics) |
| CLARIN / VLO | Linguistic data centers and corpora |
| MappingSupport GIS servers | US ArcGIS Server REST roots (bounded list; skip unscoped US ArcGIS) |
| FGDC Service Status Checker | US federal/state geospatial service hosts |
| Geoseer | Indexed OGC services with layer counts |
| GeoNode gallery | GeoNode installations (gallery leftovers still yield a few live hosts) |
| ArcGIS Instant Apps Filter Gallery / org galleries | Public /apps/instant/{template}/?appid= apps (one record per public appid, not the gallery chrome) |
Vendor “customers” and “community” pages are useful but noisy: skip demos, marketing sites, and expired domains.
Hunt patterns
Recent discovery sessions (through 8 September 2026, v1.20.0) produced catalogs when they followed a named list, a software tenant pattern, or a missing type, not an unscoped web search. Copy these prompts; details live on the type pages.
| Hunt | Prompt that works | Start from | Accept | Reject |
|---|---|---|---|---|
| Software instance | Which {software} catalogs are missing? | Vendor gallery / hostname pattern / crt.sh for SaaS / GitHub forks and code search for open-source products | Live public UI matching that software.id | Demos, marketplace tenants, a second copy of the same hub (including the same catalog on github.io and a custom domain); repeating a hunt from the last two weeks |
| National harvest sources | Which data sources harvested by {national portal} are missing? | Harvest / organisations / catalogues API on the national site | Independent origin catalog (CKAN, GeoNetwork, Hub, agency /opendata list) | XML dataset feeds, price files, slices of the same national catalog, IR dumps already registered |
| Country university IRs | There are a lot of {country} universities… Which scientific repositories are missing? | OpenDOAR, ROAR, re3data country, OpenAIRE Graph | Public IR that lists datasets (DSpace Dataset type, Dataverse, research-data community) | Publication-only IRs, theses-only with no Dataset type, login walls, microstates with no universities |
| Country indicators | Which {country} indicators catalogs are missing? | NSO site, health ministry, SDG, central bank | Queryable table DB or indicator explorer | PDF publications, CMS home, IMF NSDP already registered, agency PxWeb already on the national StatBank |
| Named directory | Which catalogs from {list URL} are missing? | One bounded URL (ODIS, CoreTrustSeal, STAC Index, WIS2 GDC, GeoNode gallery, Instant Apps Filter Gallery) | Live catalog UI / harvestable API | Preservation systems with no dataset listing, org homepages, hijacked hosts |
| Subnational coverage | Which {country} cities and counties have geoportals that are missing? | Municipal GIS product tenant list for that country | Local owner in {CC}/{ISO-3166-2}/ | Google every city; REST of an existing Hub; “all 400 counties”; another PL/CZ/SK/IT/JP commune sweep after v1.20.0 |
| Custom-software review | Review custom {type} catalogs for new software definitions | Cluster remaining software.id: custom by hostname/path | New software.id when ≥3 independent installs or a first-party product page names it | Unnamed one-off .gov roots; Leaflet-only guesses; keep those custom |
Do more of: vendor tenant lists, harvest-source dumps, country shape (missing type), named directories, crt.sh for SaaS hostname patterns. Do less of: Google for “missing catalogs in the world”, university IR hunts for Monaco-class stubs, guessed HCI/Virtual LMI hostnames, repeating a software hunt from this week, city-by-city geoportal sweeps in countries whose municipal GIS product is already tenant-complete.
A hunt that finds 0 missing catalogs is complete — report that and stop. Agent recipes: agents/discover.md. Priority queue: agents/improve.md.
Identify the software
Choose software.id from data/software/ (or custom if unknown). See software-taxonomy.md and catalog-types.md.
| Platform | Typical signals | Probe (GET, timeout, public only) |
|---|---|---|
| CKAN | generator meta ckan, ckan-footer-logo, /api/3/action/package_list | https://host/api/3/action/status_show |
| DKAN | CKAN-compatible API plus /api/1/search | same as CKAN, plus /api/1/search |
| EKAN | ekan_theme / profiles/contrib/ekan, Drupal 9+ DKAN-1 successor | /jsonapi/dataset/dataset or /data.json |
| Socrata | /api/views, *.socrata.com or SODA | https://host/api/views.json?limit=1 |
| OpenDataSoft | /api/explore/v2.1/catalog/datasets | that path |
| GeoNetwork | /srv/api, /srv/eng/csw | CSW GetCapabilities |
| GeoNode | /api/layers/, /api/datasets/ | /api/layers/ |
| GeoServer | /geoserver/ows, /geoserver/rest | WMS GetCapabilities |
| MapTiler Server | Title MapTiler Server, /admin, port 3650 | /api/maps/{id}/style.json (e.g. streets) |
| MapServer | /cgi-bin/mapserv, WMS GetCapabilities mentions MapServer | WMS GetCapabilities |
| QGIS Server | qgis_mapserv.fcgi, GetCapabilities mentions QGIS Server | WMS GetCapabilities |
| mviewer | mviewer JS, /apps/*.xml config | Viewer URL plus public layer/theme config |
| Trimble Locus IMS | /IMS/, tekla-mvc-common | Public karttapalvelu tenant |
| Sitowise Louhi | OpenLayers + partner=stw | Public karttapalvelu without /IMS/ |
| Landfolio | portals.landfolio.com | Country cadastre map portal (not ArcGIS REST) |
| Spatial Suite | SpatialMap browserdetect.js?ver= | Municipal webkort |
| KortInfo | drift.kortinfo.net/Map.aspx Site= | One municipality tenant |
| IntraMaps Public | title IntraMaps, ApplicationEngine, project=Public | One council public project, not per module |
| Spectrum Spatial Analyst | /connect/analyst/ title Spectrum Spatial | One public tenant, not per mapcfg= |
| Exponare | /exponare/RestPublicApplication.aspx | One public tenant (not Public vs REST vs Mobile) |
| LocalMaps | /localmaps/gallery title LocalMaps Gallery | One gallery per council; skip REST on the same host |
| GEUSMAP | /geusmap/?mapname= | One record per mapname |
| GISApp | *.gisapp.ro | One city tenant (not ArcGIS REST) |
| XY Maps | maps.xymaps.com/{city}, /xymaps/Map | One public city tenant (not vendor home) |
| GeneGIS PAGIS | {comune}.servizigis.it | One municipality tenant (not vendor hub) |
| GisMaster | geoportale.sportellounicodigitale.it/GisMaster IdCliente= | One comune tenant (not Default.aspx as a second copy) |
| LDP SIT | cloud.ldpgis.it/{slug}/, LdP Viewer | One comune or union slug (not vendor hub) |
| GFMaplet | /page:s_italia:geoportale, *.prod.globogis.com | One geoportale tenant (not sportello home or Maggioli REST as a second copy) |
| HyG Mapgis | /mapgis/mapa.jsp or /mapgis9/mapa.jsp, HyG Consultores footer | One public aplicacion= (not mapgisigserver) |
| iObčina | /gisapp/Default.aspx?a= | Kaliopa municipal tenant |
| G3W-SUITE | g3w-client, /map/{group}/ | Public portal, not /admin or per-project |
| Astun iShare | “Powered by iShare”, /mymaps.aspx | UK My Maps / My House (not Cadcorp, not INDEPTH iShare) |
| Cadcorp SIS WebMap | SIS WebMap / GeognoSIS | Public web map (not disy Cadenza) |
| Hexagon M.App Enterprise | /Apps/, Hexagon Geospatial | Public Apps portal, not per-app |
| CubeWerx CubeSERV | /cubewerx/cubeserv GetCapabilities | Public service root, not per layer |
| Sentinel Hub | *.sentinel-hub.com STAC /api/v1/catalog | Catalog API, not EO Browser as a second catalog |
| DataWarehousePro | app.datawarehousepro.com/go/ | One tenant portal |
| Goal Tracker | *.goaltracker.org | Country SDG tenant |
| PxStat | PxStat.Data.Cube_API / “PxStat Open Data Platform” | Collection API (not PxWeb /api/v1/) |
| DGBAS Web | /DgbasWeb/ Taiwan local stats | County/city query UI; skip nstatdb |
| TabNet | title “TabNet Win32”, deftohtm.exe, cgi-bin/dh | Brazilian DATASUS tabulator |
| FENIX | fenixservices.fao.org, FAOSTAT API, CountrySTAT UI | One FENIX app (FAOSTAT, AMIS, …), not a FAO CKAN dump |
| SparkMap | sparkmap.org / All Things state hubs | One public hub (not CARES HQ as a second copy) |
| eDatos | *.edatos.io, /indicators/v1.0/indicators | Public hub (not the NSO CMS home) |
| Cancer-Rates.info | cancer-rates.info/{st}/ public query UI | Skip login-walled tenant paths |
| Conduent HCI | “Healthy Communities Institute” HTML | Public community indicator home |
| Virtual LMI | *.virtuallmi.com, /vosnet/ | State LMI home, not generic /analyzer |
| Beyond 20/20 | ReportFolders/reportFolders.aspx | Public report/cube catalog |
| RDF Online Repository | *.revenuedev.org | Country license portal |
| ResourceContracts | resourcecontracts.org | Hub or country contract catalog |
| Guangxi Public Data Open Platform | {city}.data.gxzf.gov.cn | One city or provincial tenant |
| Isogeo | OpenCatalog /api OpenAPI (not IsiGéo) | /api or public OpenCatalog home |
| openEO | landing api_version + /collections + /processes | /.well-known/openeo or API root |
| Mapbender | /application/, Mapbender viewer | HTML or title mentions Mapbender |
| gvSIG Online | /gvsigonline/, select_public_project | Page title or footer gvSIG Online |
| deegree | /deegree-webservices, CSW/WMS XML mentions deegree | OGC GetCapabilities |
| NetGIS Server | /keos/, /Netgis7 | Page title NetGIS Server 7; optional wms.ashx GetCapabilities |
| Sampaş WebGIS | /KentrehberiApp/Index | Page title contains SAMPAŞ WEBGIS |
| GiSoftGis | /GiSoftGis/#/cityguidepublic | Angular SPA; gi-ajax-loading-indicator; meta “Kent Rehberi Uygulaması” |
| Visor Urbano | visorurbano.{city}.gob.mx, {city}.visorurbano.com | Title Visor Urbano; skip visorguadalupe.com |
| Dobles Visor de Mapas | /comun/js/leaflet.js, Leaflet.GoogleMutant | Costa Rican municipal visor; title Visor de Mapas |
| GeoNube | geonube.com.ar/visor/{slug} | Leaflet/bootleaf; Cambalache; custom domains OK |
| Geopixel Cidades | {city}.geoportal.geopixel.com.br | Next.js /_next/; confirm city; wildcard DNS |
| Geodados SaaS | {city}.geodados.com.br/Publico | OpenLayers; acessoAnonimo; skip staff login root |
| MapMap | {city}.mapmap.com.br/geo-portal | Laravel citizen hub; skip cidadao PWA and Licença Suspensa |
| CTMGEO SigWEB | {city}.ctmgeo.com.br/mapa/ | Title SIGWeb; not a DNS wildcard; skip generic SIGWeb |
| DRZ WebGIS | webgis.drz.com.br/{city}/ | OpenLayers; title WebGIS/WebGis; nginx 404 for unknown slugs |
| dmCity | web.dmcity.fi/{city}/public/ | Title dmCity Web App; not generic Experience Builder |
| InfoGIS | www.infogis.fi/{municipality}/ | /codebase-infogis/; Infokartta Oy; not Louhi/IMS |
| Geo-IT GIS | geoitgis.geo-it.be/touchviewer/ | Title Geo-IT GIS™ Touch Viewer; not generic MapGuide |
| Zeljko GIS | zeljko-gis.com Fusion / zopcina.zeljko-gis.com/{tenant}/ | Not generic MapGuide; one tenant, not a second cadastral UI |
| SIGimWeb | /sigimweb/ or /sigim/, title SIGimWeb | /gomap_web/; not generic MapGuide; not GOnet or JP Cadrin CIF |
| Geocentriq | app.geocentriq.com/mrc/{mrc}, geocentriq.min.js | One MRC tenant, not per municipality; not SIGimWeb, GeoCentralis, or GOnet |
| GeoCentralis | portail.geocentralis.com/public/sig-web/{mrc}/{code}/ | One MRC or city slug, not per geo-code; not Geocentriq or SIGALE |
| SIGALE | sigale.ca/Main.aspx?mrc={code} | One MRC code, not per municipality; not Geocentriq or GeoCentralis |
| NetGIS Runtime | /NetGISRuntime/basis/index.jsp | Title NetGIS - © WSP Danmark; not Turkish netgisserver |
| ArcGIS Experience Builder | experience.arcgis.com/experience/, jimu-core/init.js, Länsstyrelsen ext-webbgis | Not dmCity tenants; not /apps/webappviewer/; not /apps/instant/ |
| ArcGIS Web AppBuilder | /apps/webappviewer/index.html?id= | Not Experience Builder; not Instant Apps |
| ArcGIS Instant Apps | /apps/instant/{template}/index.html?appid= | Not Experience Builder; not Web AppBuilder; not Dashboards |
| ArcGIS Dashboards | /apps/dashboards/{item-id} | Not Instant Apps; not Hub; not arbitrary dashboards outside /apps/dashboards/ |
| Argenmap | src/js/app.js plus IGN Argenmap modules | Not generic Leaflet |
| WebEWID | title WebEWID / Portal Mapowy | Not authenticated role portals; not EWMAPA geoportal2.pl |
| GAUSS WebCity | {org}.gis.ba, /webcity/, dist/mapstore2.js, logo_gauss.png | Not generic MapStore; not VertiGIS WebOffice WebCity |
| Pozi | {council}.pozi.com | Title Pozi Web Map; not IntraMaps or Exponare |
| JMap | /JMapWeb/ or JMap NG /services/ng/ | Not hostnames that merely contain jmap |
| GIS Cloud | {city}.giscloud.com | Not MuniSight login |
| MRF Web Map | {county}.mrf.com / js/lib/mrf/ | Not web.munisight.com (munisight) |
| MuniSight | app.munisight.com/{Tenant} or {tenant}.gis.catalisgov.ca Catalis Login.aspx | Not mrf; not generic GeoMedia WebMap |
| GeoMedia SmartClient Public Maps | publicmaps.gisquadrat.com/BP/WEPM.aspx ig.publicmaps | Not GeoMedia WebMap $GP; not ERDAS APOLLO |
| SIT WebGis | webgis.sit-puglia.it/{comune}/ ng-app="WebApp" or SIT- | Not Regione Puglia GeoNetwork; not Lizmap |
| p.mapper | /pmapper/ or {city}.geo-portale.it | Not UMN MapServer as the public catalog |
| Weave | title Weave Map webpack app.*.js | Not GeneWeaver; not IntraMaps or Pozi |
| eKMap Cloud | assets/ekmapboxgl/ekmap-mapboxgl.js | Not quyhoach.hanoi.gov.vn Next.js; not Vinh Phuc OL or HCMC VLAB |
| OVIE | /js/libs/OpenLayers/OL.js + Materialize | Not Mission Viejo geoviewer.io; not INEGI /mdm6/ (mxsig) |
| SOFTPRO | {city}.cadastre.com.ua / SOFTPRO /js/locale/ua.js | Not kadastr.gov.ua or map.land.gov.ua |
| MxSIG | /mdm6/ or /mxsig2/ amplify.js | Not OVIE |
| cardo | /net3/public/, cardo.Map | Public map/catalog under /net3/public/ |
| GC Navi | geocloud.jp/webgis/ | Tenant WebGIS home |
| NOL-IS | maps.nol-is.de / static.nol-is.de | Public municipal geoportal |
| map.apps | /mapapps/ | con terra viewer; catalog UI may be smartfindersdi |
| MapSolution | /MapSolution/apps/home/welcome, title Home MapSolution | IP SYSCON viewer; not con terra mapapps |
| CoGIS | CoGIS Portal | Portal home; elitegis only if that is the branded viewer |
| OpenGeoPortal | federated layer search | Geoportal home, not a single layer |
| Knoema | *.knoema.com or branded hub | Portal home only — not every dataset URL |
| StatPlanet | title StatPlanet, data.csv / StatPlanet Cloud | Public explorer home, not a single indicator URL |
| SDMX-RI | NSIWebService / NSIStdV20Service | Public NSI/SDMX catalog |
| GENESIS-Online | genesis-online.css or gor-webapp; /genesis/online or /datenbank/online | Table catalog (POST-heavy API) |
| IBIS-PH | /ibisph-view/, IBIS-Q | Public indicator home |
| eDatos | /indicators/v1.0/indicators, *.edatos.io | JSON-stat or public ODS UI |
| Cancer-Rates.info | cancer-rates.com/{code}/ | Public query UI, not login |
| Conduent HCI | “Healthy Communities Institute” | Public indicator home |
| Virtual LMI | *.virtuallmi.com, /vosnet/ | Public LMI home |
| DHIS2 | /api/system/info, /dhis-web-commons/ | JSON version from system info; skip login-only HMIS |
| IPUMS | *.ipums.org, extract UI | Collection home (USA, International, CPS, …), not a single extract |
| OpenAIRE | explore.openaire.eu, *.openaire.eu CONNECT gateway | EXPLORE or national/community gateway home, not a single research product |
| Yoda | yoda., public landing + DataCite | Institutional public dataset landing, not the login vault |
| Our Open Data | /assets/cms/public.css | Catalog home, not a numeric dataset page |
| DataPress | CKAN API plus DataPress chrome | Prefer datapress over ckan when branded |
| Haplo | Haplo research repository | Public outputs/data catalog |
| FAIRDOM-SEEK | FAIRDOMHub / WorkflowHub | Investigations/catalog home |
| RAMADDA | RAMADDA repository UI | Folder/catalog entry |
| ICAT | facility data catalog | Public search or documented REST/OAI |
| BelsisIMS KRH | ims.*/Projects/*/Pages/KRH.aspx | ASP.NET KRH city-guide; do not confuse with Netcad Netigma |
| VertiGIS WebOffice | /synserver, /WebOffice/synserver, wo-hosting.vertigis.com, map.geoportal.at | Page title VertiGIS WebOffice; weboffice_packed.css; core/flex/mobile clients |
| VertiGIS Studio Web | /vertigisstudio/web/?app=, /gcx/WebViewer/?app=, /Geocortex/WebViewer/?app=, apps.vertigisstudio.com, apps.vertigisstudio.eu | #gcx-app; GA title VertiGIS Studio Web; not Html5Viewer or Essentials REST sites |
| Geocortex Essentials | /Geocortex/Essentials/REST/sites, /Html5Viewer/, *.geocortex.com | Title Geocortex Essentials Sites Directory or Geocortex Viewer for HTML5; licensed Geocortex footer |
| GeoMedia WebMap / Geospatial Portal | /geoportal01/, /cdngiportal/, /msip/Full.aspx, /Online_Mapping/ | Version: + Licensed to:; Intergraph.WebSolutions / $GP.; title may be Geospatial Portal or GeoMedia WebMap Publisher Portal |
| disy Cadenza | /cadenza/, /public/, /pages/map/, /fachauswertungweb/ | cadenza/disy in HTML; Cadenza Web or Workbooks UI; guest login plus theme/workbook navigator |
| ArcGIS Server | /rest/services, /arcgis/rest/services | /rest/info?f=pjson |
| ArcGIS Hub | /api/search/v1, portal sharing REST | /api/search/v1 |
| Idra | /IdraPortal/, /Idra/api/v1/ | Federation UI or REST JSON; usually catalog_type: Data search engine |
| FAIR Data Point | RDF DCAT at /, fdp-client, /swagger-ui | GET with Accept: text/turtle or application/ld+json |
| Aristotle MDR | Aristotle registry of data elements / vocabularies | Public registry home or /api/v4/ |
| Fusion Registry | SDMX structural metadata, Fusion Registry branding | SDMX REST catalog |
| Metadata Browser | MetadataWorks catalog UI | Public browser home |
| Wagmap / わが街ガイド | www2.wagmap.jp, GeoAccessJS | Tenant map gallery |
| SonicWeb | www.sonicweb-asp.jp/{slug}/, sonicweb.js | Tenant map home |
| GeDA-Public | *.geogeo.jp | Tenant eマップ home |
| Geolonia スマートマップ | {org}.tottori-geomap.jp, Next.js GIS | Tenant map UI (not Kazakhstan smartmap) |
| EWMAPA | *.geoportal2.pl, GEOBID | Public municipal SIP |
| GeoMapFish | ngeo / gmf-, /themes JSON | Theme API plus map UI |
| Tianditu | 天地图 / tianditu | Public province or city node |
| MapGIS IGServer | /igs/rest/mrcs/docs, /igs/rest/services | IGS REST catalog; not ArcGIS on a mapgis.* host |
| Masterportal | Masterportal config / LGV viewer | Public layer tree |
| WIS2 Box | wis2box, pygeoapi | OGC API or discovery UI |
| Liferay (open data only) | RISP / datos abiertos module | Dataset listing, not a CMS homepage |
| oPortal | /oportal/ | Government catalog home |
| OGD Platform India | data.gov.in tenant | Ministry/state catalog |
| Elsevier Digital Commons | bepress / Digital Commons | IR root + OAI |
| InstDB | FairStack / InstDB | Institutional node home |
| WEKO3 | WEKO3 IR | Repository root |
| OBiBa Mica | Mica / OBiBa study catalog | Public /ws/ or study search |
| Dataverse | /api/dataverses, /api/info/version | /api/info/version |
| DSpace | /oai/request?verb=Identify, /handle/ | OAI-PMH Identify |
| Invenio / Zenodo-like | /api/records | /api/records?size=1 |
| OPUS | /oai?verb=Identify, OPUS 4 UI | repository-root /oai?verb=Identify |
| RADAR | /radar/de/home, /oai/OAIHandler, /radar/api/datasets | OAI Identify and datasets JSON with totalHits |
| Symbiota | /collections/datasets/rsshandler.php, “Powered by Symbiota” | Collection search and/or dataset RSS |
| CONTENTdm | *.contentdm.oclc.org, /digital/api/collections | that path and/or /oai/oai.php?verb=Identify |
| Omeka S | /api JSON-LD, /api/items | /api/items |
| Fedora Repository | /fcrepo/rest or /rest | Fedora version headers; prefer Hyrax/Islandora/PHAIDRA if that is the public UI |
| PHAIDRA | /api/oai, /api/search/select | OAI Identify |
| Esploro | *.esploro.exlibrisgroup.com or campus /esploro | Institutional research-outputs portal |
| Converis | Clarivate Converis CRIS | Public datasets module, not publications/persons |
| Omega-PSIR | Omega-PSIR / “Baza Wiedzy” | University CRIS root (Poland) |
| TalkBank | *.talkbank.org, CHAT / CLAN | One bank home (CHILDES, AphasiaBank, …), not a transcript |
| CLLD | *.clld.org, clld-static | One project app, not a language or parameter page |
| LabKey Server | begin.view, Panorama Public | Public project/study home, not a single assay run |
| Synapse | synapse.org, Sage Bionetworks | Portal or project catalog, not a syn######## file |
| XNAT | /xnat/, /data/projects | Public project catalog, not a session |
| OMERO | /webclient/, IDR | Repository home, not a single image |
| Kadi4Mat | /api/records, Kadi4Mat chrome | Public instance, not a login-only lab tenant |
| TR32DB | /site/index.php + /lib/js/generic/helper.js | Not CRC806DB (crc806db.uni-koeln.de) |
| e!DAL | e!DAL / IPK Gatersleben | Public DOI catalog, not a single landing |
| NOMAD | /prod/v1/api/v1/info | Oasis or central archive, not a calculation entry |
| InterMine | /begin.do, /service/version | One mine, not intermine.org or a gene report |
| GRIN-Global | /gringlobal/ | National/centre install, not the vendor homepage |
| PlutoF | api.plutof.ut.ee, PlutoF workbench | Workbench/DOI catalog — not UNITE |
| JGI Genome Portal | MycoCosm / Phytozome / PhycoCosm | Portal organism catalog, not IMG/GOLD/data.jgi.doe.gov |
| cBioPortal | /api/info portalVersion | Instance study list, not a single study view |
| ESA Science Archive | TAP /tap/capabilities | One mission archive, not a FITS file |
| ODWeb | path /odweb/ | City/provincial catalog, not the parent CMS |
| OpenGov | {org}.opengov.com /transparency | Budget reports; skip Hub sites named OpenGov |
| IMF NSDP | “National Summary Data Page”, SDMX XML | Country NSDP page, not DSBB or a Knoema wrapper |
| PxWeb | /api/v1/ | /api/v1/ |
| NADA | /index.php/api or microdata UI | site home + API path |
Browser checks that help without an API:
- Page footer (“Powered by …”)
- HTML generator meta tags
- Network tab: calls to
/api/3,/srv/api,/rest/services robots.txtand/sitemap.xmlsometimes list API paths
Cross-check at least two signals before setting software.id. If nothing matches, use custom rather than guessing.
URL patterns worth trying
Only request public URLs. Use a short timeout. Stop on 401/403 — do not attempt to bypass authentication.
Open data
/api/3,/api/3/action/package_search/data.json,/catalog.json,/catalog.xml(DCAT)/api/explore/v2.1/catalog/datasets(OpenDataSoft)/IdraPortal/and/Idra/api/v1/(Idra)/oportal/(Inspur oPortal)/openinf/(Seoul Open Data Plaza)/assets/cms/public.css(Our Open Data)*.revenuedev.org(RDF Online Repository tenant)/contract/resources(ResourceContracts){city}.data.gxzf.gov.cn(Guangxi public-data tenant)/odweb/(ODWeb municipal/provincial catalog){org}.opengov.com/transparency(OpenGov financial reports)
Geospatial
- GeoNetwork:
/geonetwork/srv/eng/csw?SERVICE=CSW&VERSION=2.0.2&REQUEST=GetCapabilities - GeoServer:
/geoserver/ows?service=WMS&version=1.3.0&request=GetCapabilities - MapServer:
/cgi-bin/mapserv?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities - QGIS Server:
/cgi-bin/qgis_mapserv.fcgi?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities - mviewer:
/apps/config XML (oftendefault.xml) - Isogeo:
/apiOpenAPI on an OpenCatalog host - openEO:
/.well-known/openeoor/openeo/landing JSON - gvSIG Online:
/gvsigonline/ - NetGIS Server:
/Netgis7(version title) and/keos/(public KEOS viewer) - Sampaş WebGIS:
/KentrehberiApp/Index(titleSAMPAŞ WEBGIS) - GiSoftGis:
/GiSoftGis/(city-guide hash#/cityguidepublic) - BelsisIMS:
/Projects/.../Pages/KRH.aspxon anims.host - cardo:
/net3/public/ - GC Navi: tenant on
geocloud.jp/webgis/ - map.apps:
/mapapps/ - MapSolution:
/MapSolution/apps/home/welcome - VertiGIS WebOffice:
/synserveror/WebOffice/synserver(titleVertiGIS WebOffice) - VertiGIS Studio Web:
/vertigisstudio/web/?app=or/gcx/WebViewer/?app=(#gcx-app/ titleVertiGIS Studio Web) - Geocortex Essentials:
/Geocortex/Essentials/REST/sites?f=pjsonor/Html5Viewer/(titleGeocortex Essentials Sites Directory/Geocortex Viewer for HTML5) - Trimble Locus IMS:
/IMS/withimscore/tekla-mvc-commonbundles - Sitowise Louhi: OpenLayers
/Scripts/integration/openlayers/plus Sitowisepartner=stw - Landfolio:
portals.landfolio.com/{country}/ - Spatial Suite:
/js/standard/browserdetect.js?ver=(SpatialMap) - KortInfo:
drift.kortinfo.net/Map.aspx?Site= - IntraMaps Public:
/intramaps90/default.htm?project=or{council}.spatial.t1cloud.com/spatial/intramaps/ - Spectrum Spatial Analyst:
/connect/analyst/mobile/withmapcfg= - Exponare:
/exponare/RestPublicApplication.aspxor/exponare/publicinvoker.aspx - LocalMaps:
/localmaps/gallery(Eagle Technology NZ galleries) - GEUSMAP:
/geusmap/?mapname= - GISApp:
{city}.gisapp.ro - XY Maps:
maps.xymaps.com/{city}or/xymaps/Map - Pozi:
{council}.pozi.com(title Pozi Web Map) - JMap:
/JMapWeb/or JMap NG/services/ng/ - GIS Cloud:
{city}.giscloud.com - MRF Web Map:
{county}.mrf.comorjs/lib/mrf/ - MuniSight:
app.munisight.com/{Tenant}or{tenant}.gis.catalisgov.caCatalis Login.aspx - GeoMedia SmartClient Public Maps:
publicmaps.gisquadrat.com/BP/WEPM.aspx(ig.publicmaps.application.min.js) - SIT WebGis:
webgis.sit-puglia.it/{comune}/(ng-app="WebApp"+ol.js, or AngularSIT-{TOWN}) - p.mapper:
/pmapper/or{city}.geo-portale.it - CommunityView:
VECommunityView/cities/{city}/ - MS-GIS:
{city}.msgis.net - Weave: HTML title
Weave Map(Cohga webpackapp.*.js) - eKMap Cloud:
assets/ekmapboxgl/ekmap-mapboxgl.js(not Hanoiquyhoach.hanoi.gov.vn) - MapMap:
{city}.mapmap.com.br/geo-portal(Laravel citizen hub; not{city}.cidadao.mapmap.com.br) - OVIE:
/js/libs/OpenLayers/OL.js+ Materialize (INEGI economic GIS) - SOFTPRO:
{city}.cadastre.com.uaor/js/locale/ua.jsUkrainian MBK - MxSIG:
/mdm6/or/mxsig2/amplify.js Mapa Digital - ArcGIS Instant Apps:
/apps/instant/{template}/index.html?appid= - ArcGIS Dashboards:
/apps/dashboards/{item-id} - Argenmap:
src/js/app.jsplus IGN template (not Leaflet alone) - WebEWID: title
WebEWID/ Portal Mapowy on*.webewid.pl - GeneGIS PAGIS:
{comune}.servizigis.it - GisMaster:
geoportale.sportellounicodigitale.it/GisMaster/GisMaster/VisualDesc.aspx?IdCliente= - LDP SIT:
cloud.ldpgis.it/{slug}/or “LdP Viewer” - GFMaplet:
/page:s_italia:geoportaleor{comune}.prod.globogis.com - HyG Mapgis:
/mapgis/mapa.jsp?aplicacion=or/mapgis9/mapa.jsp?aplicacion=(HyG Consultores footer; not/igs/rest/) - iObčina:
/gisapp/Default.aspx?a= - G3W-SUITE:
/map/{group}/{project}/or/api/ - Astun iShare:
/mymaps.aspxor/myhouse.aspx - Cadcorp SIS WebMap: public SIS / GeognoSIS viewer
- Hexagon M.App Enterprise:
/Apps/ - CubeWerx:
/cubewerx/cubeserv?SERVICE=WMS&REQUEST=GetCapabilities - Sentinel Hub:
https://services.sentinel-hub.com/api/v1/catalog/1.0.0/ - disy Cadenza:
/cadenza/or/pages/map/default/index.xhtml(Cadenza Web / Workbooks) - ArcGIS:
/arcgis/rest/services?f=pjson - GeoMapFish:
/themes - MapGIS IGServer:
/igs/rest/mrcs/docs?f=jsonor/igs/rest/services?f=json - Wagmap:
https://www2.wagmap.jp/plus tenant path - SonicWeb:
https://www.sonicweb-asp.jp/plus tenant path - GeDA-Public:
{city}.geogeo.jpor{city}.e-map.geogeo.jp - Geolonia スマートマップ:
{org}.tottori-geomap.jp(distinct from Kazakhstansmartmap) - WIS2 Box / pygeoapi: OGC API landing page
Scientific and metadata
- OAI-PMH:
/oai/request?verb=Identifyor/oai?verb=Identify - Dataverse:
/api/info/version - RADAR:
/radar/api/datasetsand/oai/OAIHandler?verb=Identify - Symbiota:
/collections/datasets/rsshandler.php - Breedbase:
/brapi/v2/serverinfo - ESGF:
/esg-search/search - CONTENTdm:
/digital/api/collections - Omeka S:
/api/items - FAIR Data Point: catalog root with
Accept: text/turtle - Digital Commons: repository root or
/do/oai/ - Fusion Registry: SDMX REST catalog
- Knoema: branded hub home (not a dataset page)
- Converis: public CRIS / datasets listing (stop on
/wskeys) - Omega-PSIR: university “Baza Wiedzy” home
- TalkBank:
{bank}.talkbank.orgcollection home - CLLD:
{project}.clld.orgplusclld-static - PxStat:
/public/api.restful/PxStat.Data.Cube_API.ReadCollection/ - TabNet:
deftohtm.exe/cgi-bin/dhquery form - FENIX: FAOSTAT
/faostat/api/v1/groups-and-domains - SparkMap: public hub home (not CARES HQ)
- LabKey:
begin.view/ Panorama Public project - Synapse: public project catalog (not
syn########) - XNAT:
/data/projectsor/xnat/data/projects - OMERO:
/webclient/or/api/v0/m/projects/ - Kadi4Mat:
/api/records - TR32DB:
/site/index.php+/lib/js/generic/helper.js - NOMAD:
/prod/v1/api/v1/info - InterMine:
/begin.door/service/version - GRIN-Global:
/gringlobal/ - PlutoF:
https://api.plutof.ut.ee/v1/ - JGI:
https://genome.jgi.doe.gov/portal/ - cBioPortal:
/api/info - ESA Science Archive:
/tap/capabilitiesor/tap-server/tap/capabilities - IMF NSDP: country page titled “National Summary Data Page”
Generic
/api,/api/docs,/swagger.json,/openapi.json
After the record exists, scripts/apidetect.py can fill endpoints[] for known platforms:
python scripts/apidetect.py detect-single catalogdatagov --dryrun
python scripts/apidetect.py detect-software ckan --dryrun
Drop --dryrun only when you intend to write YAML. Prefer --action insert so existing endpoints are kept. Full command list: apidetect.md. URL reachability of link is a separate report: liveness.md.
Automated helpers in this repository
| Tool | Use |
|---|---|
python scripts/sync_ckan_ecosystem.py --dry-run | CKAN sites from ecosystem.ckan.org; then sync without --dry-run into data/scheduled/ |
python scripts/extract_openaire_portals.py list-sources --output /tmp/openaire_sources.json | OpenAIRE Graph data sources; then match-registry and add-scheduled --dry-run (openaire-sync.md) |
python scripts/builder.py add-single URL --scheduled | Create one YAML from a verified URL |
python scripts/apidetect.py detect-single catalogdatagov | Probe known API paths on an existing record (apidetect.md) |
python scripts/check_liveness.py --sample 10 | Probe link reachability (liveness.md) |
python scripts/re3data_enrichment.py enrich --dry-run | Fill _re3data when a re3data identifier is present |
CKAN sync details: ckan-sync.md. OpenAIRE Graph sources: openaire-sync.md. Re3Data: re3data.md.
Do not write internet-wide scanners in this repository. Vendor/government lists, documented search-engine queries, and targeted GETs against candidate hosts are enough. How to query Google, Censys, and similar indexes: discovery-search-tools.md.
One catalog per public product
The same hostname often runs several GIS products. Register one YAML per public catalog UI, not per backend.
| You see | Keep | Do not also add |
|---|---|---|
GeoNetwork (or CSW) + GeoServer /geoserver | The catalog (GeoNetwork / CSW) | A second GeoServer record for the same layer list |
| Lizmap, QWC2, or mviewer + QGIS Server / MapServer | The viewer (lizmap / qwc2 / mviewer) | qgisserver or mapserver on that host |
| ArcGIS Hub + ArcGIS Server REST on the same org | Hub if it is the public catalog; Server if REST is the product | Both for the same layer set |
| STAC API + STAC Browser | stacserver when the API is public | stacbrowser on the same origin |
| openEO + STAC on one API | openeo | A second stacserver row |
MapGIS /igs/rest/ on a host named mapgis.* | mapgisigserver | arcgisserver unless the path is /arcgis/rest/ |
HyG Mapgis /mapgis/mapa.jsp + ArcGIS REST | hygmapgis viewer | arcgisserver or mapgisigserver on the same host; a second Mapgis row where GeoNetwork is already the catalog |
VertiGIS Studio Web ?app= + ArcGIS REST or Html5Viewer | vertigisstudioweb viewer | Extra ?app= GUIDs on the same tenant; do not retag Essentials /Html5Viewer/ as Studio Web |
| GET SDI / GeoMapFish / CoGIS wrapping GeoServer | The portal software | Bundled GeoServer |
| G3W-SUITE + QGIS Server | g3wsuite portal | qgisserver on the same host |
| Landfolio map portal + ArcGIS REST | landfolio cadastre UI | REST already tagged arcgisserver |
GISApp city viewer + webadaptor.gisapp.ro REST | gisapp tenant | ArcGIS REST as a second city catalog |
XY Maps maps.xymaps.com/{city} + www.xymaps.com/{city} | One public city tenant | The other SaaS hostname; Geocortex/ArcGIS Eckersall built for the same city |
GeneGIS PAGIS city SIT + {comune}.servizigis.it | The public municipal tenant | Vendor hub, cloud alias of the same comune, IndexPC.aspx as a second copy of Index.aspx |
| LDP SIT Drupal + LdP Viewer | ldpgis municipal SIT | Drupal as software; MapCreator as a second catalog of the same comune |
Maggioli STU geoportale + cartografia*.maggioli.cloud REST | gfmaplet geoportale page | REST as a second catalog of the same comune; GisMaster on sportellounicodigitale.it |
KortInfo Site= + SpatialMap webkort | The public viewer the kommune documents | Both IDs unless they are distinct public apps |
| Sentinel Hub STAC + EO Browser | sentinelhub catalog API | stacserver or Browser as a second catalog |
| ArcGIS Hub + Instant Apps / Experience Builder / Dashboards / StoryMaps / Server on the same org | The public catalog UI the owner documents (usually Hub or the named viewer) | A second Esri product for the same layer set unless it is a distinct public catalog |
| Instant Apps Filter Gallery + each gallery item | One Instant App per public appid that is itself a catalog | The Filter Gallery chrome as a second catalog; every widget/template |
Finnish /IMS/ vs Sitowise Louhi | trimblelocus if /IMS/; else louhi | Both IDs on the same karttapalvelu |
Duplicate-check link and the service origin before add-single. Harvest follows the same grain: harvest-geoportals.md.
Verify before adding
- Open the homepage. Confirm it is a catalog (search, dataset list, map layers, or metadata records).
- Confirm
catalog_typefrom catalog-types.md. If a site is both a map viewer and an open-data portal, pick the primary product. - Confirm country and, for regional/local owners, the subregion folder (
US-CA/,FR-IDF/, …). - Set
status: activeonly if the site responds. Useinactivefor dead sites you still want to record; usescheduledfor unverified finds. - Record working
endpointswhen you have them. Leaveapi: falseif you did not find a public API. - Add the record with
add-single(recommended) or a YAML file whose filename matchesid. Then follow agents/contribute.md / CONTRIBUTING.md:assign,validate-yaml --id {id}.
Conduct
- Respect
robots.txtand site terms. Public catalog metadata only. - Space out requests (about one to two seconds between hosts is enough for manual work).
- Do not collect personal data, credentials, or non-public APIs.
- Do not treat a single CSV download page as a catalog unless it is clearly a catalog UI or harvestable endpoint.
Related
- discovery-search-tools.md — Google, Censys, Shodan, FOFA (Censys alternative), URLScan, crt.sh
- discovery-agent-tools.md — Cursor, ChatGPT, Claude, MCP, and API setup
- discovery-opendata.md / discovery-geoportals.md (SDI, viewers) / discovery-scientific.md (domain) / discovery-metadata.md / discovery-indicators.md / discovery-other.md
- software-index.md — every
software.id→ recipe - apidetect.md / liveness.md
- agents/discover.md — agent checklist (hunt types)
- agents/improve.md — what to hunt next
- agents/contribute.md — write YAML after a find
- catalog-types.md
- software-taxonomy.md
- harvest.md — datasets inside a registered catalog (not this discovery job)
- cli.md