Tuesday, April 03, 2012

Did I Say 32 Cores? How about 64?

Remember when I blogged about linear read scalability out to 32 cores? Well, the awesome Nate Boley provided me with access to his brand new 64-core server. I ran my usual suite of read-only pgbench tests just to baseline its performance, and found that the performance scaled linearly all the way out to 64 clients. OK, it wasn't quite linear: the 64-client performance was only 63.68 times the single-client performance. Still, I'll take it. Graph is below.




Greg Smith mentioned to me yesterday that he's seen that kink between 16 and 32 cores on other tests that have nothing to do with PostgreSQL, such as STREAM, and that he believes it has to do with Linux's scheduler behavior.

For the curious, this machine is running Linux 3.2.  Previous benchmarking efforts on a different 64-core server running an earlier version of PostgreSQL 9.2devel and an older Linux kernel were not a success, but the lseek scalability changes in 3.2 seem to have made a big difference, and the newer PostgreSQL code has some changes that mitigate ProcArrayLock contention.

27 comments:

  1. simon@2ndQuadrant.comApril 03, 2012 12:29 PM

    Good news. Great work Robert

    ReplyDelete
  2. It sounds like you attribute the big performance jump in 9.2 vs. 9.1 mostly to Linux's changes. Is that correct, or did I misunderstand?

    ReplyDelete
  3. Joshua: No, I'm saying that to get good performance on a 64-core server, you're going to need PostgreSQL >= 9.2 *and* Linux >= 3.2. Most of the changes are actually on the PostgreSQL side, but the lseek scaling stuff in the Linux kernel was important, too.

    ReplyDelete
    Replies
    1. Robert, can you comment on whether such improvements will be seen on Solaris 10 or not ? Any guesses , ideas ?

      I use postgres 8.3 and it probably scales uptil 16-cores on solaris 10. If postgres 9.3 can push it up to 32 cores, I might as well switch to that.

      Thanks for this post.

      Delete
    2. Thanks for this great post.

      Can you comment on whether PostgresSQL >= 9.2 would scale as it does on linux >= 3.2 on solaris 10 ? I use postgres 8.3.7 on solaris 10, which in my experience, scales uptil 16 cores. If PostgresQL >= 9.2 scales as well as you've shown on solaris 10, I might as well switch to that.

      Delete
    3. Thomas, I have not tested on Solaris 10. It should be better than 8.3, but I don't know by how much.

      Delete
  4. Hmm, makes me wonder how FreeBSD scales with this.

    ReplyDelete
    Replies
    1. I think FreeBSD is still working on resolving some issues in the virtual memory subsystem that prevent it from scaling well at higher core counts. However, I'm hopeful that we'll see some positive news on that front later this year.

      Delete
  5. do you have results for a 3.0 or a 2.6 kernel?

    ReplyDelete
  6. Yeah, definately need to test with FreeBSD.

    ReplyDelete
  7. Chris PartridgeApril 03, 2012 7:02 PM

    Great stuff

    ReplyDelete
  8. Robert,

    Permission to use your graphs in an upcoming 9.2 presentation?

    ReplyDelete
  9. Josh, you're welcome to use them, with attribution.

    ReplyDelete
  10. Hello Robert,

    You are lucky to have access to a 64-Core server!

    Looking at your graph, PG serves 30k requests/sec. at concurrency 6.

    At this concurrency, on a mere 6-Core CPU, G-WAN (our Web application server with C, C++, D, Objective-C and Java scripts) is processing:

    Type of Request 6 Clients 1,000 Clients
    -------------------- ------------------ ------------------
    100-byte static file ...... 250k requests/sec. 720k requests/sec.
    dynamic 100-year Loan ..... 150k requests/sec. 160k requests/sec.

    http://gwan.ch/imgs/localhost_gwan2.10.8.png
    http://gwan.ch/imgs/localhost_gwan2.10.10_loan%28100%29k.png
    http://gwan.ch/en_apachebench_httperf.html#weighttp
    http://gwan.ch/source/ab.c
    http://gwan.ch/source/loan.c

    Scaling vertically is mandatory in a world of multi-Core CPUs - but performing vertically makes vertical scalability fly (much higher).

    I would be very willing to make tests with you about how G-WAN and PG can bring this game a step higher (contact me at my first name @trustleap.com).

    Sincerely,

    Pierre.

    ReplyDelete
  11. Related benchmarks: http://www.osnews.com/story/25334/DragonFly_BSD_MP_Performance_Significantly_Improved — but Linux was faster anyway

    ReplyDelete
  12. I'm not certain, but I believe each AMD 6272 processor has 16 cores. If so, the title of the chart probably should read "4 x 16-core AMD 6272 processors".

    ReplyDelete
  13. Robert

    What about FreeBSS performance scalability? Is it linear as well?

    ReplyDelete
  14. Can you publish the raw numbers for TPS vs # Clients? Something looks strange between 20 and 36 clients for the PG 9.2 Current curve.

    ReplyDelete
    Replies
    1. nclients pg91 pg92devel
      1 5126.256565 5606.254711
      2 9875.569885 10024.552448
      4 18770.215456 20058.372291
      8 38556.199226 42486.574505
      12 53542.256900 62905.209873
      16 64716.084974 83002.638514
      20 71026.444659 98034.531842
      24 74377.904306 115449.670165
      28 74436.084120 133433.767200
      32 72830.033132 173886.541815
      36 70592.447148 228249.257321
      40 68150.060935 255120.364728
      44 65814.140755 277912.139404
      48 63792.179169 297396.741853
      52 62674.478417 312949.733691
      56 60771.345728 327812.113433
      60 59543.504487 340869.668340
      64 57924.077125 354613.280879
      68 56070.311028 357854.846415
      72 55275.496822 358128.415224
      80 52899.178807 357011.082207

      Delete
  15. Oracle migration to EDB using EDB studio tool seems fine, but all data types seems not supported, is there a plan to incoperate that too in future ? so that we can simply do Oracle migration easily

    ReplyDelete
    Replies
    1. Please use http://forums.enterprisedb.com/forums/list.page for support on EnterpriseDB products and services, or consult your account manager for pricing and product plans. Unfortunately, I'm not able to comment on those issues here.

      Delete
  16. I have a 64-core freebsd server and would be willing to run the test suite on it.

    ReplyDelete
  17. Go for it Reed, i'm sure many await your results!

    ReplyDelete
  18. Hi Pierre Gauthier,
    Show us the code!

    ReplyDelete
  19. Hi Robert

    I know this post is a year old now - however I have at my disposal at 64 core IBM Power server running Linux. Would be more than happy (and interested) to see how performance compares on Power architecture - also happy to collaborate if there's anything I can use this server to help with.

    Many thanks,

    James

    ReplyDelete