Miner

Schema Miner - extract RDF schema patterns via SELECT queries.

This module runs three lightweight SELECT DISTINCT queries and assembles the schema in Python:

  1. Typed-object patterns:

    SELECT DISTINCT ?sc ?p ?oc WHERE {
  ?s ?p ?o . ?s a ?sc . ?o a ?oc .
}

  2. Literal patterns (datatype properties):

    SELECT DISTINCT ?sc ?p (DATATYPE(?o) AS ?dt) WHERE {
  ?s ?p ?o . ?s a ?sc . FILTER(isLiteral(?o))
}

  3. Untyped-URI patterns (URI objects without rdf:type):

    SELECT DISTINCT ?sc ?p WHERE {
  ?s ?p ?o . ?s a ?sc .
  FILTER(isURI(?o))
  FILTER NOT EXISTS { ?o a ?any }
}

All queries use OFFSET / LIMIT pagination via SparqlHelper.select_chunked().

The primary export is MinedSchema (-> JSON-LD). It can also be converted to downstream LinkML / SHACL / RDF-config exports.

class SchemaMiner(endpoint_url: str, graph_uris: str | list[str] | None = None, chunk_size: int = 10000, class_chunk_size: int | None = None, class_batch_size: int = 15, delay: float = 0.5, timeout: float = 120.0, counts: bool = True, two_phase: bool = True, unsafe_paging: bool = False, report_path: str | Path | None = None, filter_service_namespaces: bool = True, untyped_as_classes: bool = False, authors: list[dict[str, str]] | None = None, qlever_version: dict[str, str] | None = None, one_shot: bool = False, sparql_engine: str = '', sparql_strategy: str = '', source_name: str = '')[source]

Bases: object

Mine RDF schema patterns from a SPARQL endpoint.

Parameters:

  • endpoint_url – SPARQL endpoint URL.

  • graph_uris – Optional named-graph URI(s) to restrict queries to.

  • chunk_size – Number of rows per paginated request.

  • class_chunk_size – Page size for Phase-1 class discovery in two-phase mode. None disables pagination (single query).

  • class_batch_size – Number of classes grouped into one VALUES query in Phase-2 of two-phase mining. Default 15. Higher values send fewer queries but each query is heavier.

  • delay – Seconds to sleep between pagination requests.

  • timeout – HTTP timeout per request (seconds).

  • counts – Whether to also run COUNT queries for triple counts.

  • two_phase – Use two-phase mining (default). Phase 1 discovers all rdf:type classes; phase 2 queries properties per class. Much gentler on heavyweight endpoints like QLever/PubChem/UniProt. Pass False for the legacy single-pass strategy.

  • filter_service_namespaces – When True (the default), remove patterns whose subject, property, or object URI belongs to a service/system namespace (Virtuoso, OpenLink, etc.) from the final result.

  • untyped_as_classes – When True, treat untyped URI objects (those without an explicit rdf:type) as owl:Class references instead of the generic rdfs:Resource sentinel. Default False.

Initialize a SchemaMiner.

mine(dataset_name: str | None = None) → MinedSchema[source]

Run all queries and return a MinedSchema.

Parameters:

dataset_name – Optional human-readable name attached to the metadata.

Notes

The method also populates a MiningReport with per-phase timing, query counts, and failure stats. If a report_path was given at construction time, the JSON is flushed to disk after each phase completes.

count_instances(endpoint_url: str, graph_uris: str | list[str] | None = None, sample_limit: int | None = None, sample_offset: int | None = None, chunk_size: int | None = None, offset_limit_steps: int | None = None, delay_between_chunks: float = 20.0, streaming: bool = False, timeout: float = 120.0) → dict[str, int] | Any[source]

Count instances per class at endpoint_url.

Parameters:

  • endpoint_url – SPARQL endpoint URL.

  • graph_uris – Optional named-graph URI(s) to restrict queries.

  • sample_limit – Maximum number of classes to return.

  • sample_offset – Starting offset for pagination.

  • chunk_size – Page size when paginating.

  • offset_limit_steps – Use this value as both LIMIT and OFFSET step (overrides chunk_size).

  • delay_between_chunks – Seconds to sleep between pages.

  • streaming – If True return a generator of (class_uri, count) tuples instead of a dict.

  • timeout – HTTP timeout per request.

Returns:

{class_uri: count} dict, or a generator when streaming is True.

count_instances_per_class(endpoint_url: str, graph_uris: str | list[str] | None = None, sample_limit: int | None = None, exclude_graphs: bool = True, timeout: float = 120.0) → dict[str, int][source]

Return {class_uri: instance_count} for endpoint_url.

A simplified single-query variant of count_instances().

Parameters:

  • endpoint_url – SPARQL endpoint URL.

  • graph_uris – Optional named-graph URI(s).

  • sample_limit – Cap on the number of classes returned.

  • exclude_graphs – Unused; kept for backwards-compatibility.

  • timeout – HTTP timeout per request.

Returns:

{class_uri: count} dict.

extract_partitions_from_void(endpoint_url: str, void_graph_uris: list[str], timeout: float = 120.0) → list[dict[str, str]][source]

Query partition records from named VoID graphs.

Runs a SELECT query against each graph URI in void_graph_uris and returns the raw partition records suitable for passing to build_void_graph_from_partitions().

Parameters:

  • endpoint_url – SPARQL endpoint URL.

  • void_graph_uris – Graph URIs that are known to contain VoID.

  • timeout – HTTP timeout per request.

Returns:

List of partition dicts with keys subject_class, property, and optionally object_class / object_datatype.

generate_void_from_endpoint(endpoint_url: str, graph_uris: str | list[str] | None = None, output_file: str | None = None, counts: bool = True, offset_limit_steps: int | None = None, exclude_graphs: bool = True, dataset_uri: str | None = None, void_base_uri: str | None = None, timeout: float = 120.0) → Any[source]

Mine a VoID description from a SPARQL endpoint.

Deprecated since version Use: mine_schema() instead.

Parameters:

  • endpoint_url – SPARQL endpoint URL.

  • graph_uris – Named-graph URI(s) to restrict queries.

  • output_file – If given, serialise result as Turtle here.

  • counts – Include triple counts (passed to mine_schema()).

  • offset_limit_steps – Pagination chunk size.

  • exclude_graphs – Unused; kept for backwards-compatibility.

  • dataset_uri – Unused; kept for backwards-compatibility.

  • void_base_uri – Unused; kept for backwards-compatibility.

  • timeout – HTTP timeout per request.

Returns:

Graph with VoID triples.

mine_all_sources(sources_csv: str | None = None, *, sources: str | None = None, output_dir: str = '.', fmt: str = 'all', chunk_size: int = 10000, class_chunk_size: int | None = None, class_batch_size: int = 15, delay: float = 0.5, timeout: float = 120.0, counts: bool = True, reports: bool = True, filter_service_namespaces: bool = True, untyped_as_classes: bool = False, authors: list[dict[str, str]] | None = None, on_progress: Callable[[str, int, int, str | None], None] | None = None) → dict[str, Any][source]

Mine schemas for all sources in a JSON-LD or CSV file.

Reads a sources file (JSON-LD preferred, CSV still accepted) and runs mine_schema() for each entry whose endpoint is non-empty. Results are written to output_dir as {name}_schema.jsonld and/or {name}_void.ttl.

Per-source overrides (chunk_size, class_batch_size, timeout, etc.) in the JSON-LD file take precedence over the function-level defaults.

Parameters:

  • sources_csv – Deprecated - use sources instead.

  • sources – Path to the sources file (JSON-LD or CSV).

  • output_dir – Directory where outputs are written.

  • fmt – Export format - "jsonld", "void", or "all".

  • chunk_size – Pagination page size for SPARQL queries.

  • class_chunk_size – Page size for Phase-1 class discovery.

  • class_batch_size – Number of classes per VALUES query in Phase-2.

  • delay – Delay between paginated pages (seconds).

  • timeout – HTTP timeout per request (seconds).

  • counts – Whether to fetch triple-count queries.

  • reports – Write per-source analytics JSON reports.

  • filter_service_namespaces – Strip service/system namespace patterns.

  • untyped_as_classes – Treat untyped URI objects as owl:Class.

  • on_progress – Optional callback (dataset_name, index, total, status_or_error).

Returns:

Summary dict with keys "succeeded", "failed", "skipped".

mine_schema(endpoint_url: str, graph_uris: str | list[str] | None = None, dataset_name: str | None = None, chunk_size: int = 10000, class_chunk_size: int | None = None, class_batch_size: int = 15, delay: float = 0.5, timeout: float = 120.0, counts: bool = True, two_phase: bool = True, report_path: str | Path | None = None, filter_service_namespaces: bool = True, untyped_as_classes: bool = False, authors: list[dict[str, str]] | None = None, qlever_version: dict[str, str] | None = None, one_shot: bool = False, sparql_engine: str = '', sparql_strategy: str = '', source_name: str = '') → MinedSchema[source]

One-shot helper: mine a schema and return MinedSchema.

Parameters:

  • endpoint_url – SPARQL endpoint URL.

  • graph_uris – Named-graph URI(s) to restrict queries to.

  • dataset_name – Human-readable name for the dataset.

  • chunk_size – Pagination page size for pattern queries (single-pass and count queries).

  • class_chunk_size – Page size for the Phase-1 class-discovery query in two-phase mode. None (default) disables pagination - the class list is fetched in a single query. Set to a positive integer when the endpoint has too many classes for one response.

  • class_batch_size – Number of classes to group into a single VALUES query in Phase-2 of two-phase mining. Default 15. Higher values send fewer queries but each query is heavier.

  • delay – Delay between pages (seconds).

  • timeout – HTTP timeout per request.

  • counts – Fetch triple counts per pattern.

  • two_phase – Use two-phase mining (default True). Pass False for the legacy single-pass strategy.

  • one_shot – Run each pattern query as a single unbounded SELECT with no LIMIT/OFFSET and no fallback chain. Intended for local QLever endpoints. When True, two_phase is ignored.

  • report_path – If given, write an analytics JSON report to this path. The file is updated incrementally after each mining phase.

  • filter_service_namespaces – Strip patterns whose URIs belong to service / system namespaces (Virtuoso, OpenLink, etc.) from the result. Default True.

  • untyped_as_classes – Treat untyped URI objects as owl:Class references instead of the generic rdfs:Resource sentinel. Default False.

Returns:

Contains patterns and provenance metadata.

Return type:

MinedSchema

retrieve_void_from_graphs(endpoint_url: str, void_graph_uris: list[str], graph_uris: str | list[str] | None = None, partitions: list[dict[str, str]] | None = None, timeout: float = 120.0) → Any[source]

Build an RDF VoID graph from partition records.

If partitions are provided they are used directly; otherwise a fresh discovery query is run via discover_void_graphs().

Parameters:

  • endpoint_url – SPARQL endpoint URL.

  • void_graph_uris – Graph URIs containing VoID (used as base URI).

  • graph_uris – Unused; kept for backwards-compatibility.

  • partitions – Pre-fetched partition records.

  • timeout – HTTP timeout per request.

Returns:

Graph with VoID triples.