• xml
• json
• phps (serialized php)
• php (php code to execute)

On top of that PHP offers multiple ways to parse XML. I’m benchmarking these options to determine the most efficient decoding to implement in the next major version of Solarium, but the results should be useful for any PHP Solr implementation.

Before I get to the results some info on how I tested.
Because this is only the first of several benchmarks I want to do the next few months I needed a good benchmarking tool. I couldn’t really find one that suits, so I’ve created a tool myself. It’s inspired by PHPUnit, but instead of test-cases you write benchmark-cases. It includes concepts like annotation and dataproviders. If you know PHPunit you will probably quickly understand this example:

class SolrParserBenchmark extends Phperf\Benchmark
{

    /**
     * @return array
     */
    public function solrJsonDataProvider()
    {
        return array(
            'small-results' => array(file_get_contents(__DIR__ . '/data/results.json')),
            'big-results' => array(file_get_contents(__DIR__ . '/data/text-results.json')),
        );
    }

    /**
     * Json decode
     *
     * Test data is similar to Solr output with wt=json
     *
     * @dataprovider solrJsonDataProvider
     * @repeat 50
     *
     * @param string $data
     * @return string
     */
    public function benchmarkJsonDecode($data)
    {
        return json_decode($data);
    }
}

The tool is still very much a prototype, but I intend on improving it for use by others as soon as I find the time. The current code, including this Solr benchmark, can be found on GitHub for those interested.

Results

Each decoding method is tested for a small result (approx. 10KB) and a big result (approx. 800KB). The tests are repeated 50 times and the result is the average time.

View the results here

Conclusions

• The differences between decoding methods are quite big, in percentages
• The size of the result data has a big impact, json_decode is second fastest with a small dataset but slowest with a big set.
• There is a clear winner for both small and big datasets, unserialize
• While the differences in percentages are big, even the worst performer with the big dataset (bigger than most real world cases) only takes about 16 thousands of a second.

Based on these results the next Solarium version will probably use unserialize (phps response format).

The tests were done on a 2Ghz i7 MBP running PHP 5.3.8. As always with benchmarkts, don’t just trust my tests, but be sure to test on your own environment! Your results might vary…

So, I’ve made a simple component that allows you to slow Solr down for with an exact amount of time using the query string. When the component is enabled you can simple add your delay (in millisec) to the request like this: &delay=5000

Maybe this is useful to more people, so I’ve placed it on GitHub. Here are the steps to use it:

1. Go to the GitHub project page https://github.com/basdenooijer/raspberry-solr-plugins and download the project.
2. In the build dir of the project is one jar file. Place this in the instanceDir of your Solr core, in a folder called ‘lib’. This folder may not exist yet, in that case create it. For a Solr example install the directory is: /example/solr/lib
3. Add the two snippets below to your solrconfig.xml (I’m assuming a default solrconfig.xml, otherwise you might need to adjust the configs)
4. Restart Solr
5. Test it by adding a delay param to your request. For instance:
   http://localhost:8983/solr/select/?q=*%3A*&version=2.2&start=0&rows=10&indent=on&debugQuery=on&delay=1000
   This should delay the query by one second, you should also be able to see this in the debug output. The delay param is in milliseconds. No param or a value of 0 disables the delay.

These are the two snippets you need to add to your solrconfig.xml:

Add this inside the requestHandler definition that is named “search”:

<arr name="last-components">
<str>delay</str>
</arr>

Outside the requestHandler definition add this:

<searchComponent name="delay" class="nl.raspberry.solr.DelayComponent">
</searchComponent>

Important warning: this component is intended purely for testing purposes. It’s not advised to enable this component on a production environment. First of all it’s useless, in production you should not be testing. But it can also easily be abused to flood the server with lots of queries with a big delay.

facet=true&sort=price+asc&f.price.facet.range.gap=100&facet.range={!key%3Dpriceranges}price&hl.simple.pre=<b>&hl.fl=name,features&wt=json&hl=true&rows=20&fl=id,name,price&start=2&q=*:*&f.price.facet.range.end=1000&hl.simple.post=</b>&facet.field={!key%3Dstock}inStock&fq=price:[1+TO+300]&f.price.facet.range.start=1

And the longer one:

facet=true&sort=price+asc&f.price.facet.range.gap=100&facet.range={!key%3Dpriceranges}price&hl.simple.pre=<b>&hl.fl=name,features&wt=json&hl=true&rows=20&fl=id,name,price&start=2&q=*:*&f.price.facet.range.end=1000&hl.simple.post=</b>&facet.field={!key%3Dstock}inStock&fq=price:[1+TO+300]&fq=price:1+OR+price:2+OR+price:3+OR+price:4+OR+price:5+OR+price:6+OR+price:7+OR+price:8+OR+price:9+OR+price:10+OR+price:11+OR+price:12+OR+price:13+OR+price:14+OR+price:15+OR+price:16+OR+price:17+OR+price:18+OR+price:19+OR+price:20+OR+price:21+OR+price:22+OR+price:23+OR+price:24+OR+price:25+OR+price:26+OR+price:27+OR+price:28+OR+price:29+OR+price:30+OR+price:31+OR+price:32+OR+price:33+OR+price:34+OR+price:35+OR+price:36+OR+price:37+OR+price:38+OR+price:39+OR+price:40+OR+price:41+OR+price:42+OR+price:43+OR+price:44+OR+price:45+OR+price:46+OR+price:47+OR+price:48+OR+price:49+OR+price:50+OR+price:51+OR+price:52+OR+price:53+OR+price:54+OR+price:55+OR+price:56+OR+price:57+OR+price:58+OR+price:59+OR+price:60+OR+price:61+OR+price:62+OR+price:63+OR+price:64+OR+price:65+OR+price:66+OR+price:67+OR+price:68+OR+price:69+OR+price:70+OR+price:71+OR+price:72+OR+price:73+OR+price:74+OR+price:75+OR+price:76+OR+price:77+OR+price:78+OR+price:79+OR+price:80+OR+price:81+OR+price:82+OR+price:83+OR+price:84+OR+price:85+OR+price:86+OR+price:87+OR+price:88+OR+price:89+OR+price:90+OR+price:91+OR+price:92+OR+price:93+OR+price:94+OR+price:95+OR+price:96+OR+price:97+OR+price:98+OR+price:99+OR+price:100+OR+price:101+OR+price:102+OR+price:103+OR+price:104+OR+price:105+OR+price:106+OR+price:107+OR+price:108+OR+price:109+OR+price:110+OR+price:111+OR+price:112+OR+price:113+OR+price:114+OR+price:115+OR+price:116+OR+price:117+OR+price:118+OR+price:119+OR+price:120+OR+price:121+OR+price:122+OR+price:123+OR+price:124+OR+price:125+OR+price:126+OR+price:127+OR+price:128+OR+price:129+OR+price:130+OR+price:131+OR+price:132+OR+price:133+OR+price:134+OR+price:135+OR+price:136+OR+price:137+OR+price:138+OR+price:139+OR+price:140+OR+price:141+OR+price:142+OR+price:143+OR+price:144+OR+price:145+OR+price:146+OR+price:147+OR+price:148+OR+price:149+OR+price:150+OR+price:151+OR+price:152+OR+price:153+OR+price:154+OR+price:155+OR+price:156+OR+price:157+OR+price:158+OR+price:159+OR+price:160+OR+price:161+OR+price:162+OR+price:163+OR+price:164+OR+price:165+OR+price:166+OR+price:167+OR+price:168+OR+price:169+OR+price:170+OR+price:171+OR+price:172+OR+price:173+OR+price:174+OR+price:175+OR+price:176+OR+price:177+OR+price:178+OR+price:179+OR+price:180+OR+price:181+OR+price:182+OR+price:183+OR+price:184+OR+price:185+OR+price:186+OR+price:187+OR+price:188+OR+price:189+OR+price:190+OR+price:191+OR+price:192+OR+price:193+OR+price:194+OR+price:195+OR+price:196+OR+price:197+OR+price:198+OR+price:199+OR+price:200+OR+price:201+OR+price:202+OR+price:203+OR+price:204+OR+price:205+OR+price:206+OR+price:207+OR+price:208+OR+price:209+OR+price:210+OR+price:211+OR+price:212+OR+price:213+OR+price:214+OR+price:215+OR+price:216+OR+price:217+OR+price:218+OR+price:219+OR+price:220+OR+price:221+OR+price:222+OR+price:223+OR+price:224+OR+price:225+OR+price:226+OR+price:227+OR+price:228+OR+price:229+OR+price:230+OR+price:231+OR+price:232+OR+price:233+OR+price:234+OR+price:235+OR+price:236+OR+price:237+OR+price:238+OR+price:239+OR+price:240+OR+price:241+OR+price:242+OR+price:243+OR+price:244+OR+price:245+OR+price:246+OR+price:247+OR+price:248+OR+price:249+OR+price:250&f.price.facet.range.start=1

Normally a range would be used instead of that very long filterquery, but it’s an easy way to create a long request URI.

Each query is executed 100 times, and the total time measured. And this is repeated 25 times to check for consistency. Both scripts are tested this way once with GET and once with POST.

The results:

• 100 short query GET requests take an average total time of 0.16 seconds. No noticeable difference by using POST.
• 100 long query GET requests take an average total time of 0.22 seconds. No noticeable difference by using POST.
• The timings are very consistent

So, performance is not really a factor in this. How about logging? This depends on your setup, but with a default Solr setup the logging for a query received by GET or POST is the same.

Conclusion

Using GET or POST is doesn’t really make a big difference in a standard Solr setup.
However a POST request will usually have a higher limit, depending on your servlet container. In Jetty the default ‘headerBufferSize’ is 4kB. Tomcat has a similar setting ‘maxHttpHeaderSize’, also 4kB by default. This limit applies to all the combined headers of a request, so it’s not just the querystring. This might seem more than enough, but I have already seen two cases where a query with many filterqueries and facets reached this limit.
In comparison, the default for POST data in tomcat (‘maxPostSize’) is 4MB. Jetty uses a ‘maxFormContentSize’ setting with a lower default value of 200kB, but still way higher than the header limit and more than enough for even the biggest queries.

So, if you plan on constructing very big queries POST might be a safe choice. In other cases a GET request will be just fine.
And, there is always the option to simple raise your servlet container limits instead of switching to POST.

While writing this post I’ve started to wonder if moving query parameters to the Solr config by using a custom request handler would make a difference. I’ll test that in a new post soon.

If you take into account that I’m not a frontend developer (I strongly prefer backend work) I think the result is quite ok. I’ve even made a logo! The site will be further improved in several areas over the coming months, but it’s already a big step up from the previous wiki based site. The wiki is still in use for the manuals, as it works perfectly for writing documentation.

The new website also includes a blog. From now on I will post all Solarium related blog entries on the Solarium website. Next step for Solarium is completing the docs for 2.0. I’ve already made a start on it, but still need to write lots of docs.

When I started to add new features I quickly discovered that it was no option to add all new functionality to the existing select query object. It would become an unmanageable and inefficient class with 100+ methods. As a result I created a component structure. The query object only has an API for the Solr common query parameters, all other functionality is in component classes. This is comparable to how Solr works. So there is a MoreLikeThis component, a Highlighter component etcetera. Each component is only loaded when used, so Solarium is not slowed down by features you don’t use.

There is one downside to this solution though, it breaks backwards compatibility as I moved faceting into a ‘FacetSet’ component (because facets are not part of the common query parameters). The changes required to get existing code working with this new structure are relatively small, but it’s not compatible with 1.0. As a minor release may not break compatibility this requires a new major release, 2.0.

In the meantime I got several questions from people using Solarium how to add features they needed. Solarium 1.0 does not really offer a good way of extending or modifying it. Something as simple as added a few custom params to the request is currently not supported.
This was never a design goal for 1.0, but I realise people will need this and Solarium would also need it in the future to add optional features like debugging or caching. Not all features can be added to the main code, this would create way to much overhead.

Since a new major release was already needed for the query components I’ve decided to resolve these issues at the same time, so I wouldn’t need to do a 3.0 release shortly after 2.0. These are the changes currently planned for Solarium 2.0:

1. Adjust query flow. Currently this process is not very suitable for customization. The flow will be altered in such a way that it’s possible to customize all parts of the flow.
   Because the flow is currently completely internal to Solarium and not directly available to users this has no impact. The external interface will be the same as in 1.0.
2. The client object will become more of a central manager. Solr communication is already done by the adapters, now all related settings will also be moved to the adapters. This way the client object only has the role of main API access point.
   Impact for 1.0 users is limited, some connection settings need to be moved to the adapter.
3. Add a plugin system for all important concepts. This includes querytypes, requestbuilders , responseparsers and query components. Existing code will also be refactored into this structure.
   This only adds new features, no impact for 1.0 users.
4. Add an event-hook system. All important phases of the flow get a ‘pre’ and ‘post’ event, with the possibility to modify data. This can be used by end-users for doing things like custom params, and by Solarium for adding optional features like debugging without slowing down Solarium.
   This only adds new features, no impact for 1.0 users.
5. Add query components, as described above.
   Impact for 1.0 users limited to faceting, facet methods need to be called in a slightly different way.

I will describe the changes in more detail later, but they might still change during development. The focus of the changes is on improving the structure for future development, while limiting the impact for 1.0 users to only a few settings / methods.

To demonstrate the benefits of the new structure here are some use cases possible in 2.0:

• Use only the query API of Solarium and handle the request in your own code. This can make sense if you already have a good solution in place and only want to replace a string-based query builder with the Solarium query API.
• Add custom params or headers to the Solr request. You can still use the normal API, but easily customize the request using the preExecuteRequest event-hook. (names of the event-hooks are yet to be determined)
• Add a custom querytype
• Add a custom query component
• Cache some parts of the Solarium flow to optimize performance

Development has already started, but will need some time. I hope to have a working prototype within a few weeks, but a lot of work will also need to go into testing and documenting. Development will be done in the branch ‘feature/new-structure’.

First of all I decided to make good use of phpdoc, a no-brainer. This way I can generate API docs and Phpdoc works great with IDE features like autocompletion and inline documentation.

But API docs alone are not enough, somekind of manual was needed for background info, examples, guidelines etcetera. I’ve seen or worked with multiple solutions in the past, ranging from word docs to wikis, custom websites and docbook. Some were easy to rule out (like Word docs…) but this still left me with several options with their own pros and cons.

I started out by using the wiki option of the GitHub project. The big advantage is that you can enable it with the click of a button and only need to add some content. No hosting, no setup, no development. But after creating several pages I realised there are several drawbacks:

• You are very limited by what the wiki syntax offers, no custom scripts or html allowed
• GitHub wikis don’t support Gists (strangely enough)
• As with most wikis there is no way to create a page hierarchy
• And several other small annoyances

So I started searching for another solution. After some research I ended up with Sphinx or DocBook. Both use markup files for generating docs in multiple formats. I really liked the idea of making the documentation part of the github project files, and exporting into multiple formats.
I tried DocBook. While this looked ideal and the result was ok, editing DocBook was tedious. I could not find a good DocBook WYSIWYG editor, and XML editing is very time consuming. And after editing I needed to ‘build’ the XML into an output format to check the result.

After making little progress I decided this was not the way to go. Writing documentation is never fun, but this was really tedious. So, again, I decided to look for an alternative. This time with ease of use as a top priority. I want to be able to easily write documentation, and anyone else wanting to help should be able to do so without a big learning curve.

The best experiences I had with documenting were with wikis. They allow for easy editting, can be customized and allow for collaborative editing. The biggest drawback of wikis is lack of structure, the pages are only loosely related. In most wikis you need to find your own way by searching for keywords or following links in the content. But besides from that issue, a wiki would be good solution.

So I started with a basic mediawiki install. Added a skin, and some settings to disable several unwanted options. Next, I needed to add some structure for creating a manual. I found the hierarchy extension that seemed ok. It did require some fiddling to get working, but nothing too complicated.
After some testing I found several issues, but I knew I could get it working. With some customization I’ve created a solution that I’m very happy with. You can see the index on the wiki homepage and navigation on all manual pages, like this one.
I think it’s the best of both worlds: I have a structured manual that is easy to maintain, and still have the freedom of a wiki for all other pages.

I did add some other small options to the wiki. The GitHub extension was installed (and slightly modified to support inclusion of a specific gist file) to included Gists, this works much nicer than editing inline code examples. And I created a template for API doc integration. Finally I added the ambox template for nice alerts in the manual.

With the wiki in place the writing of documentation speeded up. For the first time I actually enjoyed writing docs and while writing docs I also came across several issues in the code that I didn’t notice before, similar to how writing unittests can point to design issues. Writing documentation forces you to describe the working of your code in detail, exposing any unlogical parts.

While it still needs more work I think the current result is quite nice, you can see it at http://wiki.solarium-project.org/.

So, I’ve created a poll. The most requested features will be placed at the top of the roadmap.

[I'm sorry, the poll has been closed by now...]

To demonstrate the differences I’ve done some simple benchmarks with three different update strategies, and as you will see the performance difference can be huge. I will also give some tips on how to easily optimize the updates in your application.

The benchmark scripts were built in PHP with Solarium. I’ve left out the setup part in the tests, see the Solarium wiki for more info about using Solarium.
I used two systems on a local network, one running Solr and one running the PHP client. This adds some network latency, so results for a local Solr might vary.
The Solr index has just over 100K documents. Each of the test scripts will add 1000 documents to this index.

First test: adding and committing each document

This test commits each single document. A very common scenario. Normally this would be spread out over time, but for the benchmark I do a thousand add/commits in a loop:

$start = microtime(true);

for ($id = 8000000; $id < 8001000; $id++) {
    $document = new Solarium_Document_ReadWrite;
    $document->id = $id;
    $document->name = 'test';

    $query = new Solarium_Query_Update;
    $query->addDocument($document);
    $query->addCommit();
    $client->update($query);
}

echo round(microtime(true)-$start, 2);

Result: 55.74 seconds (18 documents per second)

Second test: adding each document, single commit

This test adds the same number of documents, but only commits once. This test is comparable in performance to a Solr setup with the ‘autoCommit’ feature (in that case you should not use the commit command)

$start = microtime(true);

for ($id = 8001000; $id < 8002000; $id++) {
    $document = new Solarium_Document_ReadWrite;
    $document->id = $id;
    $document->name = 'test';

    $query = new Solarium_Query_Update;
    $query->addDocument($document);
    $client->update($query);
}

$query = new Solarium_Query_Update;
$query->addCommit();
$client->update($query);

echo round(microtime(true)-$start, 2);

Result: 12.04 seconds (83 documents per second)

The performance difference is caused by Solr having to do only a single commit instead of a thousand. This test issues the same number of Solr requests (actually +1 for the commit) so network latency should be the same.

Third test: adding and committing all documents in a single request

The final test combines the complete update into a single Solr request. While the commands are now issues in a single request, it is still the exact same set of commands as used in the second test.

$start = microtime(true);

$query = new Solarium_Query_Update;
for ($id = 8002000; $id < 8003000; $id++) {
    $document = new Solarium_Document_ReadWrite;
    $document->id = $id;
    $document->name = 'test';
    $query->addDocument($document);
}

$query->addCommit();
$client->update($query);

echo round(microtime(true)-$start, 2);

Result: 0.45 seconds (2220 documents per second)

The performance difference is caused by a lower number of requests. A lower number of requests is more optimal in multiple layers. Most importantly network latency, but less requests also mean less overhead in Solarium and Solr.

How to optimize updates

As you can see from the results the performance difference can be huge. The most optimal update strategy for your application will depend on multiple factors. First of all you need to determine how you are going to update:

• batch updates, e.g. a nightly update cron or other fixed interval
• continuous (maybe even concurrent) updates, e.g. based on user input
• or maybe a combination of both

For batch updates you will probably get the best performance with a solution similar to the third test. For very big updates you might need to break it up into several requests, followed by a single commit. How to implement this depends on the Solr client library you use. As you can see it is quite easy using Solarium.

If you have continuous updates you should prevent issuing commits with each update. This can cause a high number of commits or even concurrent commits.
The easiest solution for this scenario is using the Solr ‘autoCommit’ feature. This way you can add documents without worrying about when to commit. You only need to a a single setting to your Solr config and remove commits.

Disclaimer
The results of these benchmarks are influenced by many factors: index size, document size, index schema, update frequency, hardware, network, solr configuration and many more factors. The tests are a worst-case scenario, if you use single update+commits but they are spread out enough it might not be an issue at all.
You should really run your own tests in your own environment with a realistic workload to validate the results.

In my previous post Integrating Solr with PHP I did a comparison of several of the available options. Since then I’ve done more research and started to make notes of all issues I came across and all features I missed. Based on these notes I’ve started a project that tries to accurately model Solr and go one step beyond the existing solutions.

This process has taken several months, with lots of refactoring to reach what I think is the optimal model. At first I developed it as a library for my own projects, but I’ve decided to turn it into an opensource project. The project is called ‘Solarium’ and can be found on github:
https://github.com/basdenooijer/solarium

Please see the GitHub project wiki for more info about Solarium, the current status and some examples.
I think the Solarium library in it’s current state is already very useful, but lot’s of great features are upcoming. I will regularly post updates.

For most clients these commands are actually translated into Solr XML update messages. Taking a look at the documentation in the Solr wiki (http://wiki.apache.org/solr/UpdateXmlMessages) shows all the options available. It also shows that it is possible to combine multiple commands in a single request.
So if you need to do several deletes and add some documents this can all be done in a single request. This got me wondering, how does this actually work? Are all ‘commands’ executed in the order of the XML message?

To test this I have created a very simple Solr index with just two fields:

• id (int)
• name (string)

For posting XML update messages I use Curl:

curl http://localhost:8983/solr/core0/update? -H "Content-Type: text/xml" --data-binary @data.xml

This is the content of the data.xml file:

<update>
  <delete>
    <query>*:*</query>
  </delete>
  <add>
    <doc>
      <field name="id">1</field>
      <field name="name">item 1</field>
    </doc>
    <doc>
      <field name="id">2</field>
      <field name="name">item 2</field>
    </doc>
    <doc>
      <field name="id">3</field>
      <field name="name">item 3</field>
    </doc>
  </add>
  <commit/>
</update>

The index can easily be reset by re-posting this XML. The delete all query ensures a complete cleanup.
For testing a complex update I created this XML:

<update>
  <delete>
    <id>3</id>
  </delete>
  <commit/>
  <add>
    <doc>
      <field name="id">4</field>
      <field name="name">item 4</field>
    </doc>
  </add>
  <rollback/>
  <add>
    <doc>
      <field name="id">5</field>
      <field name="name">item 5</field>
    </doc>
    <doc>
      <field name="id">6</field>
      <field name="name">item 6</field>
    </doc>
  </add>
  <delete>
    <id>1</id>
  </delete>
  <commit/>
  <optimize/>
</update>

The expected behaviour is:

1. we start with [1,2,3] in the index.
2. we delete document 3 and commit, so we now have [1,2]
3. add document 4 and rollback, so no change: [1,2]
4. add document 5 and 6, delete 1 and commit+optimize: [2, 5, 6]

The outcome: just as expected! After posting the update the index contains document 2, 5 and 6. So Solr does handle the commands in the update XML message in the exact given order.

An important note: this example was made to test Solr. In most cases where the order of the commands matters something is wrong. In this test the order only matters because of the add and rollback of document 4. But in that case document 4 shouldn’t be in the update anyway. And normally you should only commit once, below all add/delete commands.

The other parts of this test are more useful. Like using Curl for testing XML update messages and adding multiple commands to a single update message. The most obvious example is adding multiple documents in a single update, and most clients support this. But combining multiple delete queries or even combining delete queries with adding of documents is not supported by most clients, while it can save a lot of requests.
There is a downside though, if any command fails everything below it will not be executed.