用PHP解析巨大的XML文件

我正在尝试将DMOZ内容/结构XML文件解析为MySQL,但是所有现有的脚本都已经很老了并且不能很好地工作。 如何在PHP中打开大型(+ 1GB)XML文件进行解析?

Ian asked 2020-01-13T20:35:06Z
7个解决方案
81 votes

只有两个PHP API真正适合处理大文件。 第一个是旧的expat api,第二个是较新的XMLreader函数。 这些api读取连续流,而不是将整个树加载到内存中(这是simplexml和DOM所做的)。

例如,您可能需要查看DMOZ目录的部分解析器:

<?php

class SimpleDMOZParser
{
    protected $_stack = array();
    protected $_file = "";
    protected $_parser = null;

    protected $_currentId = "";
    protected $_current = "";

    public function __construct($file)
    {
        $this->_file = $file;

        $this->_parser = xml_parser_create("UTF-8");
        xml_set_object($this->_parser, $this);
        xml_set_element_handler($this->_parser, "startTag", "endTag");
    }

    public function startTag($parser, $name, $attribs)
    {
        array_push($this->_stack, $this->_current);

        if ($name == "TOPIC" && count($attribs)) {
            $this->_currentId = $attribs["R:ID"];
        }

        if ($name == "LINK" && strpos($this->_currentId, "Top/Home/Consumer_Information/Electronics/") === 0) {
            echo $attribs["R:RESOURCE"] . "\n";
        }

        $this->_current = $name;
    }

    public function endTag($parser, $name)
    {
        $this->_current = array_pop($this->_stack);
    }

    public function parse()
    {
        $fh = fopen($this->_file, "r");
        if (!$fh) {
            die("Epic fail!\n");
        }

        while (!feof($fh)) {
            $data = fread($fh, 4096);
            xml_parse($this->_parser, $data, feof($fh));
        }
    }
}

$parser = new SimpleDMOZParser("content.rdf.u8");
$parser->parse();
Emil H answered 2020-01-13T20:35:19Z
12 votes

这与在PHP中处理大型XML的最佳方法非常相似,但是提出了非常好的特定答案,解决了DMOZ目录解析的特定问题。但是,由于一般来说,这对于大型XML来说在Google上都是不错的选择,因此我也会从另一个问题中重新发布我的答案:

我对此:

[https://github.com/prewk/XmlStreamer]

一个简单的类,在流式传输文件时,它将所有子级提取到XML根元素。已对来自pubmed.com的108 MB XML文件进行了测试。

class SimpleXmlStreamer extends XmlStreamer {
    public function processNode($xmlString, $elementName, $nodeIndex) {
        $xml = simplexml_load_string($xmlString);

        // Do something with your SimpleXML object

        return true;
    }
}

$streamer = new SimpleXmlStreamer("myLargeXmlFile.xml");
$streamer->parse();
oskarth answered 2020-01-13T20:35:53Z
9 votes

最近,我不得不解析一些非常大的XML文档,并且需要一种可以一次读取一个元素的方法。

如果您具有以下文件<Object/>

<?xml version="1.0" encoding="UTF-8"?>
<Complex>
  <Object>
    <Title>Title 1</Title>
    <Name>It's name goes here</Name>
    <ObjectData>
      <Info1></Info1>
      <Info2></Info2>
      <Info3></Info3>
      <Info4></Info4>
    </ObjectData>
    <Date></Date>
  </Object>
  <Object></Object>
  <Object>
    <AnotherObject></AnotherObject>
    <Data></Data>
  </Object>
  <Object></Object>
  <Object></Object>
</Complex>

并想退回<Object/>s

PHP:

require_once('class.chunk.php');

$file = new Chunk('complex-test.xml', array('element' => 'Object'));

while ($xml = $file->read()) {
  $obj = simplexml_load_string($xml);
  // do some parsing, insert to DB whatever
}

###########
Class File
###########

<?php
/**
 * Chunk
 * 
 * Reads a large file in as chunks for easier parsing.
 * 
 * The chunks returned are whole <$this->options['element']/>s found within file.
 * 
 * Each call to read() returns the whole element including start and end tags.
 * 
 * Tested with a 1.8MB file, extracted 500 elements in 0.11s
 * (with no work done, just extracting the elements)
 * 
 * Usage:
 * <code>
 *   // initialize the object
 *   $file = new Chunk('chunk-test.xml', array('element' => 'Chunk'));
 *   
 *   // loop through the file until all lines are read
 *   while ($xml = $file->read()) {
 *     // do whatever you want with the string
 *     $o = simplexml_load_string($xml);
 *   }
 * </code>
 * 
 * @package default
 * @author Dom Hastings
 */
class Chunk {
  /**
   * options
   *
   * @var array Contains all major options
   * @access public
   */
  public $options = array(
    'path' => './',       // string The path to check for $file in
    'element' => '',      // string The XML element to return
    'chunkSize' => 512    // integer The amount of bytes to retrieve in each chunk
  );

  /**
   * file
   *
   * @var string The filename being read
   * @access public
   */
  public $file = '';
  /**
   * pointer
   *
   * @var integer The current position the file is being read from
   * @access public
   */
  public $pointer = 0;

  /**
   * handle
   *
   * @var resource The fopen() resource
   * @access private
   */
  private $handle = null;
  /**
   * reading
   *
   * @var boolean Whether the script is currently reading the file
   * @access private
   */
  private $reading = false;
  /**
   * readBuffer
   * 
   * @var string Used to make sure start tags aren't missed
   * @access private
   */
  private $readBuffer = '';

  /**
   * __construct
   * 
   * Builds the Chunk object
   *
   * @param string $file The filename to work with
   * @param array $options The options with which to parse the file
   * @author Dom Hastings
   * @access public
   */
  public function __construct($file, $options = array()) {
    // merge the options together
    $this->options = array_merge($this->options, (is_array($options) ? $options : array()));

    // check that the path ends with a /
    if (substr($this->options['path'], -1) != '/') {
      $this->options['path'] .= '/';
    }

    // normalize the filename
    $file = basename($file);

    // make sure chunkSize is an int
    $this->options['chunkSize'] = intval($this->options['chunkSize']);

    // check it's valid
    if ($this->options['chunkSize'] < 64) {
      $this->options['chunkSize'] = 512;
    }

    // set the filename
    $this->file = realpath($this->options['path'].$file);

    // check the file exists
    if (!file_exists($this->file)) {
      throw new Exception('Cannot load file: '.$this->file);
    }

    // open the file
    $this->handle = fopen($this->file, 'r');

    // check the file opened successfully
    if (!$this->handle) {
      throw new Exception('Error opening file for reading');
    }
  }

  /**
   * __destruct
   * 
   * Cleans up
   *
   * @return void
   * @author Dom Hastings
   * @access public
   */
  public function __destruct() {
    // close the file resource
    fclose($this->handle);
  }

  /**
   * read
   * 
   * Reads the first available occurence of the XML element $this->options['element']
   *
   * @return string The XML string from $this->file
   * @author Dom Hastings
   * @access public
   */
  public function read() {
    // check we have an element specified
    if (!empty($this->options['element'])) {
      // trim it
      $element = trim($this->options['element']);

    } else {
      $element = '';
    }

    // initialize the buffer
    $buffer = false;

    // if the element is empty
    if (empty($element)) {
      // let the script know we're reading
      $this->reading = true;

      // read in the whole doc, cos we don't know what's wanted
      while ($this->reading) {
        $buffer .= fread($this->handle, $this->options['chunkSize']);

        $this->reading = (!feof($this->handle));
      }

      // return it all
      return $buffer;

    // we must be looking for a specific element
    } else {
      // set up the strings to find
      $open = '<'.$element.'>';
      $close = '</'.$element.'>';

      // let the script know we're reading
      $this->reading = true;

      // reset the global buffer
      $this->readBuffer = '';

      // this is used to ensure all data is read, and to make sure we don't send the start data again by mistake
      $store = false;

      // seek to the position we need in the file
      fseek($this->handle, $this->pointer);

      // start reading
      while ($this->reading && !feof($this->handle)) {
        // store the chunk in a temporary variable
        $tmp = fread($this->handle, $this->options['chunkSize']);

        // update the global buffer
        $this->readBuffer .= $tmp;

        // check for the open string
        $checkOpen = strpos($tmp, $open);

        // if it wasn't in the new buffer
        if (!$checkOpen && !($store)) {
          // check the full buffer (in case it was only half in this buffer)
          $checkOpen = strpos($this->readBuffer, $open);

          // if it was in there
          if ($checkOpen) {
            // set it to the remainder
            $checkOpen = $checkOpen % $this->options['chunkSize'];
          }
        }

        // check for the close string
        $checkClose = strpos($tmp, $close);

        // if it wasn't in the new buffer
        if (!$checkClose && ($store)) {
          // check the full buffer (in case it was only half in this buffer)
          $checkClose = strpos($this->readBuffer, $close);

          // if it was in there
          if ($checkClose) {
            // set it to the remainder plus the length of the close string itself
            $checkClose = ($checkClose + strlen($close)) % $this->options['chunkSize'];
          }

        // if it was
        } elseif ($checkClose) {
          // add the length of the close string itself
          $checkClose += strlen($close);
        }

        // if we've found the opening string and we're not already reading another element
        if ($checkOpen !== false && !($store)) {
          // if we're found the end element too
          if ($checkClose !== false) {
            // append the string only between the start and end element
            $buffer .= substr($tmp, $checkOpen, ($checkClose - $checkOpen));

            // update the pointer
            $this->pointer += $checkClose;

            // let the script know we're done
            $this->reading = false;

          } else {
            // append the data we know to be part of this element
            $buffer .= substr($tmp, $checkOpen);

            // update the pointer
            $this->pointer += $this->options['chunkSize'];

            // let the script know we're gonna be storing all the data until we find the close element
            $store = true;
          }

        // if we've found the closing element
        } elseif ($checkClose !== false) {
          // update the buffer with the data upto and including the close tag
          $buffer .= substr($tmp, 0, $checkClose);

          // update the pointer
          $this->pointer += $checkClose;

          // let the script know we're done
          $this->reading = false;

        // if we've found the closing element, but half in the previous chunk
        } elseif ($store) {
          // update the buffer
          $buffer .= $tmp;

          // and the pointer
          $this->pointer += $this->options['chunkSize'];
        }
      }
    }

    // return the element (or the whole file if we're not looking for elements)
    return $buffer;
  }
}
Douglas Radburn answered 2020-01-13T20:36:26Z
5 votes

这不是一个很好的解决方案,只是在这里扔了另一个选择:

您可以将许多大型XML文件分解为多个块,尤其是那些实际上只是相似元素列表的文件(因为我怀疑您正在使用的文件就是这样)。

例如,如果您的文档如下所示:

<dmoz>
  <listing>....</listing>
  <listing>....</listing>
  <listing>....</listing>
  <listing>....</listing>
  <listing>....</listing>
  <listing>....</listing>
  ...
</dmoz>

您可以一次读取一个或两个单词,然后人为地包装在根级别标签中加载的几个完整的<listing>标签,然后通过simplexml / domxml(在采用这种方法时,我使用domxml)进行加载。

坦率地说,如果您使用的是PHP <5.1.2,则我更喜欢这种方法。 在5.1.2及更高版本中,可以使用XMLReader,这可能是最好的选择,但是在此之前,您将不得不使用上述分块策略或旧的SAX / expat库。 我不知道其余的人,但是我讨厌编写/维护SAX / expat解析器。

但是请注意,当您的文档不包含许多相同的底层元素时,这种方法实际上并不实用(例如,它适用于任何类型的文件列表或URL等,但无法实现) 解析大型HTML文档的意义)

Frank Farmer answered 2020-01-13T20:37:10Z
4 votes

我建议使用基于SAX的解析器,而不是基于DOM的解析器。

有关在PHP中使用SAX的信息:[http://www.brainbell.com/tutorials/php/Parsing_XML_With_SAX.htm]

Tetsujin no Oni answered 2020-01-13T20:37:34Z
1 votes

这是旧文章,但首先出现在google搜索结果中,所以我认为我根据此文章发布了另一种解决方案:

[http://drib.tech/programming/parse-large-xml-files-php]

该解决方案同时使用XMLReaderSimpleXMLElement

$xmlFile = 'the_LARGE_xml_file_to_load.xml'
$primEL  = 'the_name_of_your_element';

$xml     = new XMLReader();
$xml->open($xmlFile);

// finding first primary element to work with
while($xml->read() && $xml->name != $primEL){;}

// looping through elements
while($xml->name == $primEL) {
    // loading element data into simpleXML object
    $element = new SimpleXMLElement($xml->readOuterXML());

    // DO STUFF

    // moving pointer   
    $xml->next($primEL);
    // clearing current element
    unset($element);
} // end while

$xml->close();
Szekelygobe answered 2020-01-13T20:38:03Z
1 votes

您可以为此将XMLReader与DOM结合使用。 在PHP中,两个API(和SimpleXML)都基于同一个库-libxml2。 大型XML通常是记录列表。 因此,您可以使用XMLReader来迭代记录,将单个记录加载到DOM中,并使用DOM方法和Xpath来提取值。 关键是方法XMLReader::expand()。该方法将当前节点加载到XMLReader实例中,并将其后代加载为DOM节点。

XML示例:

<books>
  <book>
    <title isbn="978-0596100087">XSLT 1.0 Pocket Reference</title>
  </book>
  <book>
    <title isbn="978-0596100506">XML Pocket Reference</title>
  </book>
  <!-- ... -->
</books>

示例代码:

// open the XML file
$reader = new XMLReader();
$reader->open('books.xml');

// prepare a DOM document
$document = new DOMDocument();
$xpath = new DOMXpath($document);

// find the first `book` element node at any depth
while ($reader->read() && $reader->localName !== 'book') {
  continue;
}

// as long as here is a node with the name "book"
while ($reader->localName === 'book') {
  // expand the node into the prepared DOM
  $book = $reader->expand($document);
  // use Xpath expressions to fetch values
  var_dump(
    $xpath->evaluate('string(title/@isbn)', $book),
    $xpath->evaluate('string(title)', $book)
  );
  // move to the next book sibling node
  $reader->next('book');
}
$reader->close();

请注意,扩展节点永远不会附加到DOM文档中。 它允许GC清理它。

这种方法也适用于XML名称空间。

$namespaceURI = 'urn:example-books';

$reader = new XMLReader();
$reader->open('books.xml');

$document = new DOMDocument();
$xpath = new DOMXpath($document);
// register a prefix for the Xpath expressions
$xpath->registerNamespace('b', $namespaceURI);

// compare local node name and namespace URI
while (
  $reader->read() &&
  (
    $reader->localName !== 'book' ||
    $reader->namespaceURI !== $namespaceURI
  )
) {
  continue;
}

// iterate the book elements 
while ($reader->localName === 'book') {
  // validate that they are in the namespace
  if ($reader->namespaceURI === $namespaceURI) {
    $book = $reader->expand($document);
    var_dump(
      $xpath->evaluate('string(b:title/@isbn)', $book),
      $xpath->evaluate('string(b:title)', $book)
    );
  }
  $reader->next('book');
}
$reader->close();
ThW answered 2020-01-13T20:38:41Z
translate from https://stackoverflow.com:/questions/911663/parsing-huge-xml-files-in-php