benchmarking

What advice can you give me for writing a meaningful benchmark ?

I have developed a framework that is used by several teams in our organisation. Those "modules", developed on top of this framework, can behave quite differently but they are all pretty resources consuming even though some are more than others. They all receive data in input, analyse and/or transform it, and send it further. We planned...

Linux filesystem benchmarking best practices

(Not really a programming question, sorry) I'm working on benchmarking various filesystems (most importantly: ext3) with various filesystem options (for instance: noatime, relatime etc.) for specific situations on a Linux box. For raw filesystem benchmarks, I'm looking into bonnie and bonnie++. What is the most useful way to use bonn...

Benchmark problems for testing concurency

For one of the projects I'm doing right now, I need to look at the performance (amongst other things) of different concurrent enabled programming languages. At the moment I'm looking into comparing stackless python and C++ PThreads, so the focus is on these two languages, but other languages will probably be tested later. Ofcourse the c...

What is FLOP/s and is it a good measure of performance?

Hi, I've been asked to measure the performance of a fortran program that solves differential equations on a multi-CPU system. My employer insists that I measure FLOP/s (Floating operations per second) and compare the results with benchmarks (LINPACK) but I am not convinced that it's the way to go, simply because no one can explain to me...

Is there a free tool to profile the execution time of Java methods?

Hello everybody, I need to know the usage by other applications, in time of execution, of java methods of an api created in my project, does someone know any free tool to know this? Thank you very much. ...

Is there a IDE/compiler PC benchmark I can use to compare my PCs performance?

I'm looking for a benchmark (and results on other PCs) which would give me an idea of the development performance gain I could get by upgrading my PC, also the benchmark could be used to justify the upgrade to my boss. I use Visual Studio 2008 for my development, so I'd like to get an idea of by what factor the build times would be impr...

Which javascript minification library produces better results?

Between Yahoo! UI Compressor, Dean Edwards Packer and jsmin, which produces better results, both in terms of resulting footprint and fewer errors when obfuscating. ...

How can I benchmark editor productivity objectively?

I'm not sure if Vim makes me more productive compared to other editors/ide's like Eclipse for example. But somehow I get an empowering feeling when using Vim and noticed resistance to trying others editors. I've become an Vim addict. Example: As soon a I see some cool feature in an other editor I'm thinking "Vi can do that (i just have...

Linux benchmarking tools

I've been asked to evaluate a new vendor's computing system and management has requested that I do NOT use any of our existing software to evaluate the platform (believe it or not, they have some valid reasons). Anyways, I've started trying to read up on meaningful benchmark programs for server-grade computers but haven't been all that ...

What is the best way to measure execution time of a function?

Obviously I can do and DateTime.Now.After - DateTime.Now.Before but there must be something more sophisticated. Any tips appreciated. ...

Large public datasets?

I am looking for some large public datasets, in particular: Large sample web server logs that have been anonymized. Datasets used for database performance benchmarking. Any other links to large public datasets would be appreciated. I already know about Amazon's public datasets at: http://aws.amazon.com/publicdatasets/ ...

What programming language implementation benchmarks are there?

I remember from my old BASIC programming days that for extra performance, you could use a BASIC compiler (instead of the runtime interpreter), but for ultimate performance, the answer was assembly code. Today, when performance is crucial, I guess assembly language is still the ultimate optimisation answer. Going for a less extreme solut...

Credibility of benchmarking within virtual machines?

How credible are the benchmarks carried out within a virtual machine, as opposed to real hardware? Let's dissect a specific situation. Assume we want to benchmark the performance impact of a recent code change. Assume for simplicity that the workload is fully CPU bound (though IO bound and mixed workloads are also of interest). Assume t...

Is stopwatch benchmarking acceptable?

Does anyone ever use stopwatch benchmarking, or should a performance tool always be used? Are there any good free tools available for Java? What tools do you use? EDIT: Thanks for all the answers so far. To clarify my concerns, stopwatch benchmarking is subject to error due to operating system scheduling. On a given run of your prog...

LLVM vs GCC MIPS code generation, any benchmarks?

I'm interested in knowing what is "best" free/OSS compiler for MIPS code, GCC or LLVM, or is there anything even better than those? I'm interested in knowing more about fast and memory constrained generated Assembly code than code size. In other words, does llvm-opt do the job better than gcc -O3? ...

Why is a cached Regexp outperforming a compiled one?

This is just a question to satisfy my curiosity. But to me it is interesting. I wrote this little simple benchmark. It calls 3 variants of Regexp execution in a random order a few thousand times: Basically, I use the same pattern but in different ways. Your ordinary way without any RegexOptions. Starting with .NET 2.0 these do not ge...

Does Google Analytics have peformance overhead?

To what extent does Google Analytics impact performance? I'm looking for the following: Benchmarks (including response times/pageload times et al) Links or results to similar benchmarks One (possible) method of testing Google Analytics (GA) on your site: Serve ga.js (the Google Analytics JavaScript file) from your own server. Upda...

How to correctly benchmark a [templated] C++ program

Hi, < backgound> I'm at a point where I really need to optimize C++ code. I'm writing a library for molecular simulations and I need to add a new feature. I already tried to add this feature in the past, but I then used virtual functions called in nested loops. I had bad feelings about that and the first implementation proved that th...

How to measure code performance in .NET?

I'm doing some real quick and dirty benchmarking on a single line of C# code using DateTime: long lStart = DateTime.Now.Ticks; // do something long lFinish = DateTime.Now.Ticks; The problem is in the results: Start Time [633679466564559902] Finish Time [633679466564559902] Start Time [633679466564569917] Finish Time [63367946656456...

Are there benchmarks comparing the respective memory usage of django, rails and PHP frameworks?

I have to run a Web server with many services on an embedded server with limited RAM (1 GB, no swap). There will be a maximum of 100 users. I will have services such as a forum, little games (javascript or flash), etc. My team knows Ruby on Rails very well, but I am a bit worried about Rails' memory usage. I really do not want to sta...