“large scale computing cluster” • Search • The Register Forums (2025)

Re: What was with the mainframe comment?

What is the mainframe? Well, it's a very good platform but it's ancient, arcane and, more importantly, costly. The comment was right on the money, you'd have to be an insane mainframe fetishist to buy a huge, proprietary and costly box to run x86 workloads. No creative TCO or ROI can change that. The cloud is model is not about validating the mainframe model, it's about scale out computing and software that makes a large cluster of commodity hardware offer a reliable computing service to its users. Let's keep the mainframe to what it really was meant for, that is LEGACY applications for companies that find too risky to port ancient code to a different platform.

0 0

About "...adding another (or more) layer(s)..."

Just testing myself whether I'm able to sell the cloud idea to Jake....

Q1: How, exactly, does adding another (or more) layer(s) to the stack make for better computing?

The concept of cloud as I currently understand it is to offer a pool of resources (cpu/ram/storage/network interfaces/etc).

From this pool we can choose the components that are required to complete a specific job.

For example building a router requires multiple network interfaces and right amount of ram but less cpu and storage.

In contrast building an email server requires large amount of storage and ram but we can probably manage with a single network interface (not the best practice though).

Cloud infrastructure enables us to easily build a machine that fits the specific purpose thus I think it makes for better computing.

Q2: How, exactly, does adding another (or more) layer(s) to the stack make for cheaper computing?

In terms of CPU cycles I don't think it makes for cheaper computing. From the costs perspective it can save you money.

For example let's assume I work for an agile software company that makes a new software release once a month.

On the release day company's web servers are working hard as customers are downloading the new version of the software.

For the rest of the month servers are sitting mostly idle.

Being able to scale up/down when the demand goes up/down can save you money in environments where the charging is done per CPU hour (eg.Amazon EC2).

Q3: How, exactly, does adding another (or more) layer(s) to the stack make for faster computing?

Virtualisation used in cloud computing doesn't directly offer performance gains in comparison to conventional bare-metal computing.

What cloud infrastructure enables us to do differently is to increase capacity when additional computing resources are needed.

As an example when doing cpu-intesive parallel processing in cloud one can quite quickly add 10+ nodes into the cluster which together

will get the job done quicker than 1 bare-metal computer. Deploying 10 new nodes in cloud environment is 100 quicker than deploying 10 new bare-metal computers thus it saves time.

Q4: How, exactly, does adding another (or more) layer(s) to the stack make for more secure computing?

It does not make for more secure computing.

In my opinion security is the number one common concern at the moment that slows down the adoption of cloud computing.

0 0

Re: Cloud analogy

Cloud computing, we are told, offers benefits over locally hosted servers on the grounds of scale, availablity and initial costs. There are big boys and girls there who know what they are doing and we can trust our enterprise level applications and databases with them because of the massive redundancy and expertise there.

I disagree with you that my analogy is totally invalid. Flying within clouds, like using Cloud computing, is not as problem-free as we were lead to believe. It seems that the clever boys and girls can't foresee everything that comes their way and they might as well be flying in fog. We here frequently about the bouts of «turbulence» whenever something awry happens.

Furthermore, the analogy does deal with data (or passenger) loss, merely discomfort and poor planning on behalf of those in charge as well as the power of marketing over reality.

On the topics of backups, I wonder what people will do when they 30TB databases and cubes up in the Cloud. Making backups of these is an expensive business and God help us if there is an outage while trying to download a 30TB backup to a local server. My guess is that those companies with large DBs will keep them and their backups up in the Cloud, hoping that they will never have to download them.

It might be that the solution is a sort of cluster or mirroring arrangement between the Cloud and local servers, so that when the Cloud servers become unavailable, that an automatic failover kicks in. But then, what is the point of having your servers in the Cloud?

Cloud computing is the server level equivalant of outsourcing and we all know what a success story that turned out to be.

1 0

Well on the tin it says:

(Well, on the project home page anyway):

"The Apache™ Hadoop™ project develops open-source software for reliable, scalable, distributed computing.

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile (sic) service on top of a cluster of computers, each of which may be prone to failures.

The project includes these subprojects: la la la"

So don't ask me, What businesses of many kinds need is to provide the people making the decisions with accurate information about what is happening in the business now, as well as what is happening in the market the business is in. If this Hadoop thingy can help do this, then it should be useful. If it just provides another layer of obfuscation, then not so much.

0 0

Re: Human Brain 1000000x more powerful than a computer

it's the power, cooling, and interconnect on the large scale that needs work.

Very well said. When Moore's law finally runs out, the next major breakthrough will have to be in parallel coding.

A (human) brain has ~10^11 processing elements clocked at maybe 10Hz. A CPU has ~10^9 transistors clocked at ~10^9 Hz. By that measure a humble desktop CPU should be ahead of us by a few orders of magnitude. So what gives?

Well OK, a neuron is a much more complex device than a transistor, but a million times more complex? Unless it's in some sense a quantum computational element rather than a classical one (which cannot be ruled out), I don't think there's a difference of complexity of order 10^6. Surely not 10^9 (comparing against a 1000-CPU cluster).

Which leaves the dense-packed 3D interconnect in a brain, and even more so the way it is able to utilize its hardware in parallel for a new problem without any (or much) conscious organisation of how that hardware will be organised, what algorithms it will deploy.

The next huge breakthrough will have to be in problem definition technology. Something that fulfils the same role as a compiler, but working across a much greater range between the statement of the problem and the hardware on which the code will run. There are some scary possibilities. Success on this front may reveal that a decent scientific computing cluster is actually many orders of magnitude superior to ourselves as a general-purpose problem solver, and might also reveal intelligence and consciousness as emergent properties.

Jill (*) or Skynet?

(*) Jill is the most benign first AI I know of in SF. Greg Bear, "Slant".

3 0

Weird units here → #

Large scale computing just does not work like this. There are so many examples of people who think they can apply 'enterprise' ready solutions to distributed web based apps and have failed - even with basic web sites (see the france.fr debacle for an excellent example).

I mean you no ill will but have been working in this area for many years and you really do have to think in a different manner than simply transactions per seconds.

Some simple examples (some mentioned already in the article). Think in terms of mtbf of kit when you are running a farm that contains as few as 1000 machines. Hardware failures become normal events that you have to engineer for. The idea that you can accept the 4 hours downtime per two years onb your database server and still hit you 99.9% SLA does not apply.

What about backing up 100s of TB of data? A distributed model can eliminate the need for backups.

latency - these database backends also need to return your profile page, your unread messages count and so. Cache solutions need to recalculate.

Creeping death. One node in a cluster goes down, increasing the load on the other nodes and causing the whole lot to go down.

Etc., Etc.

1 0

In bioscience, bigger is sometimes better ...

THE GOOD

I supported a biotech research computing environment for >3 decades and in my experience there were occasions when the availability of bigger systems enabled new research. For example, the emergence of TiB scale RAM systems (SGI UV) enabled the de novo sequencing of very large genomes (of plants) which, up to then, was impractical. Available software was written with much smaller genomes and smaller hardware platforms in mind, and it was inefficient and wobbly, but it worked. Phew!

Also, some researchers might not attempt ambitious computational tasks if they (mistakenly) believe it's not possible, when those of us in support groups can say "of course we can try, we just need to make the box bigger".

THE NOT YET PERFECT

Inefficient code delivered with cutting edge lab equipment required unnecessarily large resource to run. Some years ago, I spec'd and installed a multi-cabinet HPC/HTC cluster solution for the initial processing of data from a cutting-edge high throughput sequencing instrument .... only for the equipment vendor to later spend a bit of time optimising the code which meant it would now run on a single quad core desktop! This is the nature of cutting-edge laboratory coupled with cutting-edge software to handle it. The cluster capacity was then swallowed up with the unexpected volume of work to analyse the avalanche of (complex) processed data.

THE BAD

A lot of open-source research-grade software is written by people who are trying to prove a research objective, and will choose language, frameworks and platform that will just get the job done. Once achieved, there is precious little time or resource available to make the workflow or the code run efficiently, or even to re-write. This means a lot of HPC/HTC cluster time is lost to inefficient software and workflows ... and researchers use each other's code a lot, so useful (if inefficient or wobbly) software rapidly spreads within a global research community.

CONCLUSION

If we were to ask science to progress more slowly, and spend more time and money on software engineering, we could make much better use of existing hardware, but I end my comment with a defence of science that it would be wrong to do this in many cases. Sometimes the pace of change in science is so fast there is no time or funding available to optimise the many hundreds of apps used in research, much less to keep pace with advances in code libraries, hardware, storage and networking etc.. I feel the benefits of rapid advances in biological science often (not always) far outweigh the apparent background wastage of CPU cycles and IOs. Bigger is better if we want science to move ahead .. especially so when we need it to move fast ( remember covid? ).

1 0

Re: Clusters

@waterman

You are misinformed. There are (at least) two different types of scaling:

1) horizontal/scale-out scaling, i.e. a cluster (typically have 10.000s of cores and 100s of TB RAM)

2) vertical/scale-up scaling, i.e. a single large fat server (typically have 16-32 sockets and 8 TB RAM) in a SMP style

The SGI servers have always been clusters. Even SGI confirms this. Whenever you see a server larger than 16/32 sockets, you know you are dealing with a cluster. Clusters are only used for cluster workloads, such as HPC number crunching workloads or other parallel workloads. Clusters are typically serving a single user, a scientist that starts up a workload running for hours/days. All the large Linux servers having 10.000s of cores out there, are clusters. Clusters are very very cheap, you basically pay the hardware cost. If you buy a cluster with 100 cpus, it costs 100 x 1 cpu.

SMP servers are very very expensive and difficult to make. In this category you have large Unix servers and Mainframes. IBM 32 socket Unix P795 server for the old TPC-C world record, costed $35 million. Yes, one single server. To create a huge server with as many as 16 or 32 sockets, are very very difficult. Scaling up is very difficult, and that is why these servers are very very expensive. Why do you think huge SGI servers with 10.000 cores are cheap, and 16 socket Unix servers are very very expensive? You could buy many clusters for the price of one large Unix SMP server. SMP servers are typically serving many 1000 of users, running business software. If you look at the SGI customers on their homepage, none of them are using SGI servers for business ERP software, all Altix/UV servers are used as clusters. For isntance SAP HANA is clustered software.

Clusters can not replace SMP servers on business workloads. That is the reason there is always a market for huge Unix 16/32 socket servers, including Mainframes. There have never been a Linux vendor selling 16/32 SMP servers. Such large Linux SMP servers have never been for sale, they simple have never existed. You are invited to link to a Linux server with 16 or 32 sockets. Good luck finding one. Linux can not scale to 16/32 socket servers, because the hardware has never existed. Of course, people have compiled Linux onto large Unix servers, such as IBM P795 / HP Unix Integrity 64 socket server (google Linux Big Tux) - but the scaling is awful. Nobody buys a huge Unix 16 socket server for many millions to run Linux.

In short, SGI clusters are never used for SMP business workloads, just look at the use cases. They are exclusively used for clustered workloads.

SGI and ScaleMP (another Linux vendor with 10.000s of cores) explains this best. Both of these servers use a single image Linux kernel. Here they say that SGI and ScaleMP are only used for clustered workloads, and their huge Linux servers can not replace a huge 16 socket Unix server (because clusters can not be used for SMP workloads):

http://www.realworldtech.com/sgi-interview/6/

"...The success of Altix systems in the high performance computing market are a very positive sign for both Linux and Itanium. Clearly, the popularity of large processor count Altix systems dispels any notions of whether Linux is a scalable OS for scientific applications. Linux is quite popular for HPC and will continue to remain so in the future,

...

However, scientific applications (HPC) have very different operating characteristics from commercial applications (SMP). Typically, much of the work in scientific code is done inside loops, whereas commercial applications, such as database or ERP software are far more branch intensive. This makes the memory hierarchy more important, particularly the latency to main memory. Whether Linux can scale well with a SMP workload is an open question. However, there is no doubt that with each passing month, the scalability in such environments will improve. Unfortunately, SGI has no plans to move into this SMP market, at this point in time..."

http://www.theregister.co.uk/2011/09/20/scalemp_supports_amd_opterons/

"Since its founding in 2003, ScaleMP has tried a different approach. Instead of using special ASICs and interconnection protocols to lash together multiple server modes together into a SMP shared memory system, ScaleMP cooked up a special software hypervisor layer, called vSMP, that rides atop the x64 processors, memory controllers, and I/O controllers in multiple server nodes....vSMP takes multiple physical servers and – using InfiniBand as a backplane interconnect – makes them look like a giant virtual SMP server with a shared memory space. vSMP has its limits.

...

The vSMP hypervisor that glues systems together is not for every workload, but on workloads where there is a lot of message passing between server nodes – financial modeling, supercomputing, data analytics, and similar parallel workloads. Shai Fultheim, the company's founder and chief executive officer, says ScaleMP has over 300 customers now. "We focused on HPC as the low-hanging fruit "

.

Here is another discussion on huge scale-out clusters can never replace 16 socket SMP servers for enterprise business workloads:

https://news.ycombinator.com/item?id=8175726

"I'm not saying that Oracle hardware or software is the solution, but "scaling-out" is incredibly difficult in transaction processing. I worked at a mid-size tech company with what I imagine was a fairly typical workload, and we spent a ton of money on database hardware because it would have been either incredibly complicated or slow to maintain data integrity across multiple machines. I imagine that situation is fairly common.

...

Generally it's just that it's really difficult to do it right. Sometime's it's impossible. It's often loads more work (which can be hard to debug). Furthermore, it's frequently not even an advantage. Have a read of https://research.microsoft.com/pubs/163083/hotcbp12%20final.... Remember corporate workloads frequently have very different requirements than consumer."

0 0

Re: Obviously not

"...Buy an SGI Ultraviolet. Simples.

http://www.sgi.com/products/servers/uv/configs.html

Seriously - you can spec one of these with 64 Tbytes of memory...."

Well, the SGI UV2000 and the predecessor Altix servers with 64TB RAM are clusters. And they are only fit for HPC number crunching embarassingly parallel workloads. Even SGI says so. There is another Linux server as big as the SGI UV2000 server, ScaleMP sells a Linux server with gobs of ram and 10.000s of cores - and guess what? Is is ALSO a cluster. All servers bigger than 32 or 64 sockets, are clusters (sure they run a single image Linux kernel, but they are clusters, yes).

If you are going to do enterprise workloads (large databases, ERP systems, etc), then you need to go to SMP alike servers, and the largest SMP alike servers have 32 sockets (IBM P795, HP Itanium Superdome/Integrity, SPARC M6-32) or even 64 sockets (Fujitsu M10-4S).

HPC number crunching are typically running a tight for loop in the cache, on each node. So each node does not communicate too much - so this is great work for clusters such as the SGI or ScaleMP servers. Enterprise workloads, have code that branch everywhere so you need to visit all code everywhere - and for this type of workload where every cpu communicates all the time with each other - you need SMP alike servers (IBM P795, Oracle SPARC M6-32 etc). There are clustered databases running on clusters, but they can not replace a SMP database. A cluster can never replace a SMP server.

So the largest Linux servers are ordinary 8-socket x86 SMP-alike servers, and the SGI and ScaleMP clusters with 10.000s of cores and 10s of TB. There are no 32 socket Linux servers on the market, and has never been. Linux scaling is ineffecient when you go beyond 8-sockets. Sure, you can compile Linux to 32 socket unix servers - with terrible results. Google on "Big Tux", and read how Linux had ~40% cpu utilization on the HP Integrity/Superdome 64-socket server - that is really bad. (I always mix up HP Superdome and Integrity). But when you compile Linux to Unix servers, you get less optimal results. And besides, the SMP servers are very very very expensive, who would run Linux on a SMP server?? For instance, a 32 socket IBM P595 server used for the old TPC-C record costed $35 million (no typo). Who in earth would run Linux on a big SMP server?? Linux does not scale well beyond 8-sockets in the first place. It is better to buy a 10.000s of core Linux server for a fraction of the price of a large SMP server. Linux servers with 10.000s of cores are very very cheap. 16/32 socket SMP servers (Unix/Mainframes) are extremely expensive in comparison to a cheap cluster. A cluster costs you the same price as each node + a fast switch, basically. A SMP server needs lot of tailor made tech to scale to 16/32 sockets and that costs very much, because ordinary cpus does not scale.

So, there is a reason you will never see Linux in the enteprise market share: it is dominated by large SMP servers running enterpise business workloads (Unix / Mainframes), and unless Linux will scale beyond 8-scokets, there is no chance in hell Linux will venture into high end Enterpise segment - where the really big money is. Or, all Wall street banks would instead buy cheap Linux clusters with 10.000s of cores, to replace their extremely expensive 16/32 Unix / Mainframe servers.

.

SGI and ScaleMP says their largest servers, are clusters (that is, they are only used for HPC number crunching, and can not do SMP Enterprise workloads)

http://www.realworldtech.com/sgi-interview/6/

"...The success of Altix systems in the high performance computing market are a very positive sign for both Linux and Itanium. Clearly, the popularity of large processor count Altix systems dispels any notions of whether Linux is a scalable OS for scientific applications. Linux is quite popular for HPC and will continue to remain so in the future, ... However, scientific applications (HPC) have very different operating characteristics from commercial applications (SMP). Typically, much of the work in scientific code is done inside loops, whereas commercial applications, such as database or ERP software are far more branch intensive. This makes the memory hierarchy more important, particularly the latency to main memory. Whether Linux can scale well with a SMP workload is an open question. However, there is no doubt that with each passing month, the scalability in such environments will improve. Unfortunately, SGI has no plans to move into this SMP market, at this point in time..."

ScaleMP's large Linux server is also only used for HPC:

http://www.theregister.co.uk/2011/09/20/scalemp_supports_amd_opterons/

"...Since its founding in 2003, ScaleMP has tried a different approach. Instead of using special ASICs and interconnection protocols to lash together multiple server modes together into a SMP shared memory system, ScaleMP cooked up a special software hypervisor layer, called vSMP, that rides atop the x64 processors, memory controllers, and I/O controllers in multiple server nodes .... vSMP takes multiple physical servers and – using InfiniBand as a backplane interconnect – makes them look like a giant virtual SMP server with a shared memory space. vSMP has its limits....The vSMP hypervisor that glues systems together is not for every workload, but on workloads where there is a lot of message passing between server nodes – financial modeling, supercomputing, data analytics, and similar parallel workloads. Shai Fultheim, the company's founder and chief executive officer, says ScaleMP has over 300 customers now. "We focused on HPC as the low-hanging fruit..."

A developer in the comments explains:

"...I tried running a nicely parallel shared memory workload (75% efficiency on 24 cores in a 4 socket opteron box) on a 64 core ScaleMP box with 8 2-socket boards linked by infiniband. Result: horrible. It might look like a shared memory, but access to off-board bits has huge latency...."

.

If you really need true a single 32TB RAM server which is not a cluster, you need to go to Oracle SPARC M6-32 which is the only SMP alike server on the market with that much RAM. Running databases from memory will be extremely fast. (Memory database Hana is a cluster and does not count, it can not do SMP workloads as each node has to communicate with each other). Also, the new SMP alike Fujitsu Solaris M10-4S server with 64 sockets, has 32TB today and will have 64TB RAM with the new memory sticks. I think the largest IBM Mainframe has 3.5TB RAM? And the largest IBM P795 with 32 sockets has 4 (or is it 8) TB RAM? And the largest HP Superdome/Integrity has 2 or is it 4TB RAM? Matt Bryant, can you please fill us in?

0 1

Linux bad scalability

"Like the Altix UV which runs standard SUSE Enterprise Linux.

Ref :http://www.theregister.co.uk/2011/06/15/sgi_altix_sales_hadoop_prefabs/

"and expand from its own variant of SUSE Linux to a machine that can run standard SUSE Linux Enterprise Server or Red Hat Enterprise Linux as well as Microsoft's Windows Server 2008 R2....SGI has been making a lot of noise lately about how Windows Server 2008 can run on its Altix UV 100 and 1000 machines, and in fact, the UltraViolet hardware scales far beyond the limits of that Windows OS at this point. The Windows kernel sees the"

Please, dont talk about Linux scalability. It might even scale worse than Windows. First of all, there are at least, two different kind of scalabilty:

1) Horizontal scalability, scale-out. It is basically a cluster. Just add another node and the number crunching gets faster. These clusters typically have 10.000 of cores or even more. Supercomputers have many more. These clusters are used for HPC number crunching work loads, and can not handle SMP workloads.

2) Vertical scalabilty, scale-up. It is basically a single fat huge server. These huge servers, SMP-alike, typically have 16 or 32 sockets. Some even has 64 sockets. IBM Mainframes have up to 64 sockets. These costs much more than clusters. For instance, the IBM P595 with 32 sockets used for the old TPC-C record, costed $35 million list price. Can you imagine what a cluster with 32 sockets costs? Not $35 million. Probably it will cost 32 x 1 node. And if one node costs $5,000, it will be not be $35 million. These SMP-alike servers, are used for SMP workloads, typically running large databases in large configurations. HPC clusters can not do this (they can run a clustered database though, but not run a normal database).

Enterprise companies are only interested in SMP workloads (large enterprise databases etc). The reason Unix rules in Enterprise companies, is because Unix has huge SMP-alike servers capable of handling SMP workloads. Linux can not, Linux SMP servers dont exist. Linux is only used on HPC clusters, and Enterprise companies are not interested in HPC clusters.

Now regarding Linux scalability: Linux runs excellent on clusters (such as supercomputers), but scales quite bad on SMP alike servers. Linux has severe problems utilizing beyond 8-sockets. First of all, there have never existed any Linux server with 32 sockets, for sale. Recently, a couple of months ago, the Bullion released the first 16 socket Linux server. The first ever in history. And it is dog slow.

There has never ever been a 32-socket Linux server for sale. Never ever. If you know of one, please show us a link. You wont find any such a large SMP-alike server. Sure, people have compiled Linux onto IBM P795 AIX Unix server with 32 sockets - but that is not a Linux server. It is a Unix server. I could compile a C64-dos to it, and it would not make the IBM Unix server a C64. And people have compiled SuSE to HP's Unix Itanium/Integrity 64 socket server - but it is still a Unix server. They tried this before, and never sold Linux on the HP Unix server, google on "Big Tux Linux" for more information and see how bad Linux scalability it had, with ~40% cpu utilization running on 64 sockets. This means every other cpu was idling, under full load. How bad is not that?

Regarding the SGI UV1000 servers, they are clusters with 10.000 of cores. ScaleMP also has a huge Linux cluster with 10.000 of cores. It is a cluster running a software hypervisor, tricking the Linux kernel into believing it is running a SMP server - with bad scalability. Latency to nodes far away makes the cluster uncapable of handling SMP workloads. Latency on a SMP-alike server is very good in comparison, making them possible to run a large database on all cpus, without grinding to a halt.

Thus, Linux servers with 10.000 cores (that is, clusters) are not suitable of handling Enterprise SMP workloads. See yourself. The ScaleMP Linux cluster is only used for HPC number crunching:

http://www.theregister.co.uk/2011/09/20/scalemp_supports_amd_opterons/

"...Since its founding in 2003, ScaleMP has tried a different approach. Instead of using special ASICs and interconnection protocols to lash together multiple server modes together into a SMP shared memory system, ScaleMP cooked up a special software hypervisor layer, called vSMP, that rides atop the x64 processors, memory controllers, and I/O controllers in multiple server nodes....vSMP takes multiple physical servers and – using InfiniBand as a backplane interconnect – makes them look like a giant virtual SMP server with a shared memory space. vSMP has its limits.

...

The vSMP hypervisor that glues systems together is not for every workload, but on workloads where there is a lot of message passing between server nodes – financial modeling, supercomputing, data analytics, and similar parallel workloads. Shai Fultheim, the company's founder and chief executive officer, says ScaleMP has over 300 customers now. "We focused on HPC as the low-hanging fruit..."

SGI talks about their large Linux clusters with 1000s of cores:

http://www.realworldtech.com/sgi-interview/6/

"...The success of Altix systems in the high performance computing market are a very positive sign for both Linux and Itanium. Clearly, the popularity of large processor count Altix systems dispels any notions of whether Linux is a scalable OS for scientific applications. Linux is quite popular for HPC and will continue to remain so in the future,

...

However, scientific applications (HPC) have very different operating characteristics from commercial applications (SMP). Typically, much of the work in scientific code is done inside loops, whereas commercial applications, such as database or ERP software are far more branch intensive. This makes the memory hierarchy more important, particularly the latency to main memory. Whether Linux can scale well with a SMP workload is an open question. However, there is no doubt that with each passing month, the scalability in such environments will improve. Unfortunately, SGI has no plans to move into this SMP market, at this point in time..."

Ergo, you see that Linux servers with 1000s of cores, are only used for HPC number crunching, and can not handle SMP alike workloads. SGI and ScaleMP says so, themselves.

And also, there has never been a 32 socket SMP-alike Linux server for sale. Until a couple of months ago, there was no 16-socket Linux server either for sale, but Bullion released the first generation Linux SMP-alike 16-socket server. Ever. And it performs very bad, just read the benchmarks.

In comparison, Unix on 16 or 32 or 64 sockets have performed very well for decades. Linux scales well on clusters, but extremely bad on SMP-alike huge Unix servers with up to 64 sockets. The thing is, Linux developers never had access to large SMP servers, so they can not tailor Linux to such work loads. Unix devs had been able to do this for decades. In some decades from now, maybe Linux will be able to handle 32 sockets well, too. But not today. Just read SGI and ScaleMP - all them are used for HPC and avoid SMP workloads - why?

0 3

Re: Why

" Why don't SGI manufacture a big bad SMP server with 16-32 cpus instead of clusters with 1000s of cores or cpus? "

Because the technology SGI has was simply overkill; as they say in the article, they were marketing to "cloud computing" providers (which of course means once the marketing guys got out of the way they were marketing to data centers in general...) But having a multi-giga*byte*/second interconnect and so on is simply overkill when compared to (less costly) rack of systems with ethernet backing them, and SGI was unable to get costs low enough to be even vaguely competitive. Realize, virtual machines, web server software, database software, most data analysis stuff, etc., each individual process easily will fit in the amount of RAM that can fit on one blade, so the high-speed interconnect will not even come into play.

"Until SGI sells a big bad SMP server with as many as 32 cpus, I won't be impressed. Anyone can make a huge cluster with 1000s of cores and tons of RAM. "

This has been cutting *some* into SGIs sales -- throwing a pile of systems into a rack and sticking gigabit, or 10gigabit, ethernet, or Myrinet, or Infiniband, behind them, allows for a system that can run certain distributed jobs just fine and some other types "well enough". This is not what SGI has though -- they have a VERY fast, VERY low latency interconnect that permits the whole lot to act as a single system with 100s or 1000s of CPUs in it with minimal loss of performance.

"Until anyone offers a Linux server with 32 cpus for sale, I wont be impressed. "

Be impressed, 32-CPU systems shipped years ago, I've seen a few personally. IBM sells a system with 8 6-core CPUs which came out in 2010, Sun had a 4 x 8-core system already by 2010. You could buy 12-core CPUs from AMD by 2010, and run 48 cores using a 4-socket motherboard.

"The problem is that Linux can not scale beyond 8 cpus today."

It sure can.

" No one has ever offered a 16 or 32 cpu Linux server for sale."

Don't know about CPUs, but cores? They sure have.

" SGI could be the first to make Linux scale beyond 8 sockets? And then SGI could target the Enterprise market, instead of chasing HPC scientific number crunching specialized companies, running large Linux clusters."

Ignoring the first part (SGI is far from the first to scale beyond 8 sockets...), SGI tried targetting the enterprise market. As the article says, it has low margins compared to HPC. SGI's technology was overkill for this market, and so they could not get the cost low enough to be competitive.

The thing you are missing (other than insisting high-core systems don't exist, and that Linux can't use them....) is the great speed of this interconnect. ScaleMP lets you lash together a bunch of nodes and have them appear as a Single System Image; SGI's specialized chips are letting them lash together a bunch of nodes and have them appear as a Single System Image. BUT, ScaleMP is limited by the (gigabit or 10 gigabit) ethernet, or Myrinet, or Infiniband, or whatever backing it up, which at it's best has an order of magnitude higher latency than NUMALink.

NUMALink is quite close to being fast enough to actually supply the CPUs at full speed, so you can have a job spread all over and it'll still run at close to 90% efficiency. THIS is what is keeping SGI in business, there are certain computation types that break up just fine (nearly to fully independent threads, working on nearly to fully independent bits of data, will work fine on a cluster), and there are certain types that do not break up well at all (fluid dynamics and weather models to name 2.) With these types of computations, the work threads tend to each have to do somewhat of a "random walk" through the data set, and sometimes threads are pretty interdependent, both of which makes this type of job completely unsuitable for "traditional" (message passing) clusters, and very very slow on a "traditional" cluster running single system image software on it, due to the slowness (slow relative to on-board memory...) of the message-passing hardware.

2 0

Re: Switching from big iron to x86 virtualisation

@Roo

"...If your definition of x86 servers can stretch (a lot) the Cray XC-30 might be of interest. It ships with the Cray Linux Environment, a cabinet can hold 384 sockets (3072 Xeon E5 cores), infinband I/O. You can add a lot of cabinets too. At the lower end you have the SeaMicro boxes ranging from 64 to 256 sockets (and I have seen them referred to as 'servers')...."

The Cray XC-30 is a cluster. It is a HPC cluster used for number crunching. Have you seen the workloads the Cray tackles? All embarassingly parallel workloads, running on lot of nodes, on a cluster. Cray does not make SMP server (a single big fat server, running for instance databases). Cray makes computing clusters, not Enterprise servers running Enterprise work loads. You will never see such HPC clusters running big fat Oracle databases, for instance.

.

.

Anonymous Coward,

"....256 socket NUMA single system image coherent global shared memory. It is not a distributed memory HPC cluster.... The IBM P795 has 32 sockets and SuSE Enterprise Linux is one of the supported systems...."

First, ccNUMA servers are a cluster. Not SMP servers, nor close to SMP servers. They can not handle SMP workloads, they are a cluster:

http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access#NUMA_vs._cluster_computing

Second, the IBM P795 is not a Linux server. It is an AIX server that someone has compiled Linux for. I doubt anyone runs Linux on it, because the P795 is so expensive. It is better to run Linux on cheap x86 servers. Besides, Linux would never be able to scale to 32 sockets. HP had their "Big Tux" Linux server, which was the 64 socket Itanium Integrity (or was it Superdome?) server that they compiled Linux to, and Linux had something like ~40% cpu utilization using 64 sockets. Linux scaled so bad on 64 sockets, that when HP sold Big Tux, HP only allowed Linux to run in a partitioned server. The biggest supported Linux partition on Big Tux was 16 cpus. If Linux scaled well, HP would have supported 64 socket partitions too. But they didnt. 16 cpu Linux partitions did not work that well, either. If you look at modern benchmarks of Linux on a 8-socket x86 server, the cpu utilization is quite bad. For instance, SAP benchmarks show 87% cpu utilization on a 8-socket server. 16 socket would give... 60% cpu utilization, I guess. And 64 sockets does give ~40% cpu utilization in confirmed benchmarks. I am convinced the IBM P795 Linux offering is very limited in terms of Linux scalability, like, it has 40% cpu utilization, or P795 is only allowing max partitions of 8-16 cpus.

So, no, there are no Linux SMP alike servers. Some people have compiled Linux to big Unix servers, but that does not make them Linux servers. For instance, you can run Linux on a IBM Mainframe with 24 sockets, but that does not make the IBM Mainframe, a Linux server.

.

If you look at the RAM latency of a true SMP server, it has uniform latency from every cpu. No matter which cpu you are using, it accesses RAM as fast as every other cpu.

SMP alike servers, for instance the Oracle M9000 SPARC server with 64 sockets, has a worst case latency of 500ns, which is quite bad. But best case is something like 100ns or so. So the spread is tight, no big difference between worst case and best case. The Oracle M9000 is not a true SMP server, because there is some difference in latency. But that does not matter, see below.

If you look at the latency of a NUMA cluster like the SGI Altix 256 socket Linux server, the worst case latency is something like 10.000ns, or was it 70.000ns? I can't remember, but I know it was above 10.000ns, which is catastrophic. This changes everything. If you develop for the SGI server, you must allocate data close to the current node, and you design your software differently than for a SMP server - otherwise the performance will be extremely bad. In effect, you design your software exactly as if it was a cluster: you allocate data close to the current node, etc. And if you look at the workloads the customers are buying SGI for, it is for HPC workloads, and other clustered workloads. If you do SETI number crunching, each node does not have to talk to other nodes, that is a typical parallel workload that NUMA clusters handles fine.

If you study the new Oracle M6 server with 96 sockets, and ~10.000 threads, it is a mix of 8-socket SMP servers, connected with NUMA connections. But, the worst case latency is again, only 2 hops or so, just like in the Oracle M9000 server. If you need to access data, it will not take you more than 2 hops to reach it, which is fast. In effect, when developing for the M6 server, you design your program as it was a true SMP server. You dont need to allocate data to close nodes, etc, just develop your software as normal. So, Oracle M9000 and Oracle M6 runs SMP workloads just fine, and you dont have to redesign your software. Just take your current Solaris binary, and copy it to the M6 server and it will run fine. Try to do that on the Linux SGI cluster, and it will show extremely bad performance unless you redesign your program. Oracle M6 will run SMP workloads such as the Oracle Database in very large configurations. Databases is all Oracle cares about. So, in effect, the Oracle M6 and M9000 behaves as if they were true SMP servers, and you dont need to redesign your software to run on them. This is not the case using the Linux SGI clusters, they can never show good performance running large database configurations, with worst case latency of 70.000ns.

So, no, there are no Linux 32 socket servers for sale. Sure, IBM has one AIX P795 server they offer Linux on, but that does not make it a Linux server. And IBM offers Mainframe with 24 sockets to run Linux, but that does not make the Mainframe a Linux server either. If someone can show a link to Linux 32 socket server, I would be very surprised, because no one has ever manufactured such a Linux server. And NUMA servers dont count, they are just a cluster. Anyone can make a cluster, basically, just slap on lot of PCs on a fast switch and you are done.

So, I invite anyone to show a Linux server with 32 sockets. There have never been until today. Why? Linux scales very bad, shows benchmarks from HP on their HP-UX 64-socket server. Linux can not go beyond 8-socket servers today. And Linux does not even handle 8-socket servers well, just look at such benchmarks, and read about "cpu utilization".

0 0

ORIGINAL ARTICLE:

" .... Alibaba claims 'World's biggest datacenter' crown from Google

Chinese media reports that Alibaba Cloud has opened what it claims is the world’s most powerful data center.

The “Zhangbei intelligent computing center” will reportedly run AI workloads.

Alibaba Cloud has rated the facility at 12 Exaflops (12 quintillion floating-point operations per second), and claims it surpasses the nine exaflop cluster Google announced in May 2022. ...."

OUR REPLY: PFERDESHEISSE !!!

NCA's (North Canadian Aerospace - a pseudonym) northern British Columbia, Canada data centre is currently at 20 Million Square Feet inside of a very large and super-remote mountain powered by a massive methanol-based Proton-Exchange-Membrane (PEM) fuel cell setup. It takes TWO HOURS to drive in good weather using 4x4 off-road vehicles to the site it is so remotely located.

We have four massive GaAs-based supercomputers running 60 GHz and TeraHertz-scale combined-CPU/GPU/DSP/Vector Array processors in that datacenter which FAAAAAAAAAAAAAAAR OUTCLASS ANYTHING IN THE WORLD:

ThunderHorse (One YottaFLOP at 128-bits wide - 60 GHz GaAs with Inert Gas Supercooling)

DragonSlayer (One YottaFLOP at 128-bits wide - 60 GHz GaAs with Inert Gas Supercooling)

FyreScreamer (Five YottaFLOPS at 128-bits wide - Stacked Layer Cube - 2 THz GaAs on Borosilicate with Silicone Oil Immersion)

Quasar's Child (10 YottaFLOPS at 128-bits wide upgrading to 25 YottaFLOPS by March 2023 - 2 THz GaAs on Borosilicate with Silicone Oil Immersion)

They are the world's fastest supercomputers OF ANY KIND and they are ALL running WBE's (Whole Brain Emulation of Sodium, Potassium, Phosphorus, Glucose/Protein/Enzyme, etc production, uptake, attraction, repulsion, gating and pass-through to simulated neural structures) and various linear expert systems on a 24/7/365 uptime that is used for various scientific and medical-centric research and development! And soon that 20 million square feet (1,858,060 square metres) of datacentre space will DOUBLE in size by March 2023!

Google and Alibaba DON'T EVEN COME CLOSE to our systems!

NOT EVEN A TINY BIT CLOSE!

We faaaaaaaaaaaaaaaaaar outscale and outclass them by millions of times!

V

0 4

Don't share Trevor's view

The IBM I've worked with is an organization with incredible long command chains, where a small army of lawyers is dedicated to shift all the blame for anything bad done by IBM to its own customers and with another, this time much, much bigger, army of managers turned accountants who are clueless about anything their business does but only focus on their bottom line.

The very few IBMers left that do anything useful or productive (the ones that have not yet been "resource actioned" outside India) form a demoralized team that have to struggle daily with incredible tight reporting requirements and project and service workloads that span what was previously spread across many more people. All this for an increasingly small salary and benefits in the name of corporate profits they know will never get to enjoy. And make no mistake, the Indian IBMers don't share the same culture and values that made IBM what it was in the past century, they see IBM as their springboard to their next job elsewhere. One with a higher pay and better benefits.

I can't see how this can turn into the company Trevor describes. Large scale computing? Where are the potential customers? The same ones that currently feel being ripped off by an IBM that is constantly trading local people with offshored resources to increase its profits at the cost of worsening service to its customers? The same customers that bet five years ago on IBM server hardware only to see it abandoned? The same customers purchasing IBM branded storage only to discover that it is a mere repackaging of someone else's product, only with an additional layer of overworked support staff?

And where are the products? Watson? The perfect death trap for the risk averse IBM: no customer wants to sign off a contract where they will be footing the bill to set up an incredibly complex system with no ROI guarantees whatsoever.

All I can see is IBM slowly fading into irrelevance and reducing itself to two almost independent companies: one a very small, highly profitable, cluster of captive mainframe customers (these will remain IBM customers literally forever) Another made of the small army of lawyers, this time specialising in patent licensing. But this part will not last more than 30 years: on the long term, IBM can't sustain any significant R&D investment if they can't turn it into profits outside patents, something they have been unable to achieve for the last 20 years.

7 1

Re: I think the eternal question I've never really understood about the transputer is

[Author here]

There are a bunch of things that remain difficult problems, as I understand it. (I am _not_ an expert in parallel systems or anything!)

Transputers had hardware-controlled comms links, so building meshes or grids of them was relatively easy. No modern chip has that. Instead, we have processors with lots of independent cores, but it's the OS's problem to allocate tasks to cores and use them efficiently.

A little bit of that was in hardware in Transputers. There was also the Helios OS, which I've written about recently on the Reg.

https://www.theregister.com/2021/12/06/heliosng/

... and the Occam programming language, which was designed for parallel programming.

https://www.theregister.com/2022/05/06/pi_pico_transputer_code/

[Extremely simplified big-picture hand-wavy explanation]

C is not inherently parallel and does not have direct structures for this. The OS has to handle it; transputers brought that right into the language.

But Unix isn't inherently parallel either. It was built for a non-networked minicomputer with text terminals. That's why "everything's a file" and so on.

So, now, because there is so much investment in xNix and xNix code, it's being painfully added into a traditional xNix, instead of everyone moving onto Unix's appointed successor, Plan 9, or Plan 9's successor, Inferno. They have networking right in the kernel: processes can move directly from one network node to another. You can't do something like that on Linux except extremely clumsily by migrating a VM or getting Kubernetes to manage a cluster of nodes running containers: millions of lines more code on top to fix something that was built into Plan 9's kernel...

And which was done in a mixture of hardware and the programming language in Occam on Transputers.

Some of this is being reimplemented, slowly and painfully, in Go and Rust and things... in a much more difficult, complex form, in code at least 1000x bigger and more complicated and less capable and less flexible.

Nature demonstrates that the way to build a big computer is from extremely large numbers of small slow computers with lots and lots of communications links. Instead, we're building very big, hot, power-hungry computers, with no direct support for comms between them. So it's hard to communicate from one core on one chip to another core on another chip. They don't scale very well. The only way we've found so far is big clusters, with lots of computers all chewing on different bits of the same data set and not really talking to one another much.

In a lot of ways, because late-1980s OSs and languages didn't natively do hard stuff like multiprocessor support, clustering, memory protection, all sorts of stuff that FOSS xNix and Windows NT finally solved using minicomputer technology in the early 1990s, back in those days very smart people worked hard on coming up with incredibly clever fixes for the problems of parallel computing and so on.

But it was never mainstream and didn't catch on.

Then a decade later this became easy and mainstream with modernised 1970s tech, and the industry lost 15-20Y of progress and went backwards.

1 0

Re: Marketing Bull

Hi Liran,

Nice to see someone in your position actually commenting on the article.

I'm a long-time file system and storage protocol developer. I spent many years trying to solve storage problems at the file system level and I've now moved further up the stack as I believe that there are rarely any cases where high performance distributed file systems are really the answer as opposed to better designs further up the stack.

For example, the SpecSFS test is building code which is obviously quite a heavy task. I spend most of my life waiting for compiles and I would always welcome better solutions. But I already have seen huge improvements by moving away from poor languages like C and C++ towards more managed languages that have endless performance and security benefits over compiled languages.

Now, given the problem of compiling code, this has always been a heavy process. Consider that most development houses have a complete rats nest of header files dependencies in code. Simply using a library like Boost or the standard C++ library can cause decades of programmers lives to be lost. Of course the local operating system will generally RAM cache most files once they've been read once... making the file system irrelevant. But compiling something that produces a large number of object files (such as the Linux kernel) on a system which has anti-malware protection will kill performance in general.

To distribute the task of compilation across multiple systems, there are many solutions, but tools like Incredibuild handle this in a far more intelligent manor than placing a large burden on the file system. Therefore, testing file access in those regards is a meaningless solution because it presents a higher performance file system as opposed to a distributed compilation environment as the solution. Simply precompiling the headers and distributing that along with the code to be built to other systems is far more intelligent.

Then there's the case of data storage and manipulation. Your product makes a big point out of having it run side by side by with compute on large nodes which also hold storage. On algorithmic principles in terms of making file i/o perform better, making a better distributed file system that implements the POSIX APIs makes a lot of sense... if you're interested in diagnosing the symptoms but not the underlying problem.

When working with huge numbers of nodes and huge data sets, generally the data in question is structured at least in some way that can be consider object oriented. It may not be relational, but it is generally something that can be broken down into smaller computing segments.

Consider mapping a DNA strand. We could have hundreds of terabytes of data if we store more than simple ribosome classification. If we stored molecular composition of individual ribosomes, the data set will be massive. In this case, each ribosome will be able to be structured as an object which can be distributed and scheduled most intelligently in a database that handles hot and cold data distribution across the cluster through either sharding or share-nothing record replication.

Consider the storage from a collision within an LHC experiment. The data is a highly structured representation of energy readings which themselves are not structured... or at least not until we'll identified their patterns. As such, the same general principle of shared nothing database technologies make sense.

To have a single distributed file system to store this data would be quite silly as the data itself isn't well represented as a file as opposed to a massive number of database records or objects.

The only system I know of anymore where large scale file systems makes sense is virtual machine image storage. And in this case, since VMware has one of the most impressively stupid API licensing policies EVER... you can't generally depend on supporting them in a meaningful way. They actually wanted to charge me $5000 and make me sign NDAs blocking me from open sourcing a VAAI NAS driver for VMware. I simply moved my customers away from VMware instead... that was about $5,000,000 lost for them. In addition, if I had to instead a vib to support a new file system, I'd be nervous since VMware famously crashes in flames constantly due to either storage API or networking API vibs.

But that said, VM storage for Hyper-V, KVM and Xen are a great place to be. But if I'm using Hyper-V, I'll use Storage Spaces Direct, for KVM or Xen, I can see room for a good replacement for Gluster or the others.

So, now that I hit you with a book... I'm interested in hearing where your product fits.

I read your entire web page because you sounded interesting. And I found your technology to be quite interesting. Under different circumstances, I'd probably even ask for a job as a programmer to have some fun (it's sad, but I find writing distributed file systems to be fun). But I simply don't see the market segment which this technology targets. Is it meant as file storage for containers? Is there something which makes it suitable for map/reduce environments other than better database tier distribution?

I look forward to hearing back. I get the feeling you and I could have some absolutely crazy (and generally incomprehensible) conversations at a pub.

P.S. - I'm working on a system now that would probably benefit from technologies like yours if I wasn't trying to solve the problem higher up in the stack. I may still need something like this later on if you start looking towards FaaS in the future.

1 1