Boundary between "real research" and "implementation details?"

[zep]

I’m an undergraduate in computer science at University of Illinois. I am very passionate about answering unsolved research questions, and I’m am easily exasperated with wheel-reinventions that claim to be research.

I’ve been fortunate to find good advisors who have helped me to develop and contribute a few theoretical approaches that apply to a broad variety of applications. However, a huge volume of computer science journal/conference papers solely discuss implementation details and contribute little that generalizes beyond the specific application.

If much of the CS research community is focused on implementation and if I have to fight to find “real research questions” to study, is this a warning sign? Should I go be a mathematician or physicist, or am I not seeing the big picture of CS research?

If I stay in CS, what strategies can I use for finding a dissertation advisor (ideally at Berkeley, CMU, Stanford, et al.) who will be willing to focus on the big picture instead of on how I can connect software A with technology B to get a performance improvement on system C?

[the poisoned pawn]

I don't know much about system area, but for theoretical CS such as machine learning-AI, algorithm analysis, geometric computation, etc, I think almost all papers are about abstract ideas, proofs and algorithms. At the end of some paper, there are a few experimental results that demonstrate the efficiency of that particular algorithm or method.

[zep]

Pawn,

Good point. I think I need to get out of the CS systems bubble.

I don't know much about system area, but for theoretical CS such as machine learning-AI, algorithm analysis, geometric computation, etc, I think almost all papers are about abstract ideas, proofs and algorithms. At the end of some paper, there are a few experimental results that demonstrate the efficiency of that particular algorithm or method.

[fangsout]

A theory is only as valuable as its implementation. I do understand where you're coming from though, just wanted to get a jab in for all the systems peeps

[frenzydude]

Just apply for grad school in "Theory" and find profs who work in theory. You should do great.

I’m an undergraduate in computer science at University of Illinois. I am very passionate about answering unsolved research questions, and I’m am easily exasperated with wheel-reinventions that claim to be research.

I’ve been fortunate to find good advisors who have helped me to develop and contribute a few theoretical approaches that apply to a broad variety of applications. However, a huge volume of computer science journal/conference papers solely discuss implementation details and contribute little that generalizes beyond the specific application.

If much of the CS research community is focused on implementation and if I have to fight to find “real research questions” to study, is this a warning sign? Should I go be a mathematician or physicist, or am I not seeing the big picture of CS research?

If I stay in CS, what strategies can I use for finding a dissertation advisor (ideally at Berkeley, CMU, Stanford, et al.) who will be willing to focus on the big picture instead of on how I can connect software A with technology B to get a performance improvement on system C?

[gradSchoolBound]

@Zep, you have a made a historically very important point in your post. In 60s, 70s and 80s, many new ideas used to be explored around systems, and computer architecture. We can see the results of such explorations in the invention of dataflow (as opposed to Von-Neumann) machine. The ideas such as time-sharing, parallel processing, virtualization are all results of the exploration of such 'wild and crazy' ideas in systems - we have come up with virtually no new ideas in the last 30 years. Yes, some of them have been rejuvenated. At the fundamental level, most computers today are still the same.

As the computer became cheaper in the 80s and started becoming available outside the academic community and a larger part of the industry - thanks to IBM 360 machines - a huge demand for computer scientists arose. To satisfy this demand, there was a mushrooming for computer science departments all around the nation. Their goal was to meet the growing demand of the industry instead of waste several years investigating unconventional ideas without the promise of a success. There was mushrooming of computer science conferences everywhere. To mitigate the flooding of research papers in these conferences, the easiest way to verify the quality of research was to show some numbers using standard benchmarking tools. Hence, computer science research became a number game. Just having some unconventional ideas on a paper was not enough - numbers became the mantra. And one's research productivity was largely quantified by one's number of publication. Moreover, obtaining funding for exploring research idea that departed from mainstream computer science research became incredibly hard or impossible. Hence, we have the current trend of graduate research - take somebody's work, tweak it a little bit, benchmark it, obtain n times better performance and publish it. And the sad part is, a lot of these benchmarking results are wrong, as was pointed out by a conference article by a computer science professor at U of Colorado.

In some way, systems people have shot themselves in the foot. The slow-paced innovations in the systems field have contributed to the shift of research focus from fundamental computer science research in systems to its application in various fields, such as AI, bioinformatics etc, even though there are so many cool things waiting to be done in the area of systems.

Edited March 12, 2011 by gradSchoolBound

[DezB88]

I don't want to be rude, but I just don't like this kind of whining in public.

Whatever they do, let them do. Whatever you do, do it.

If you can do something great in whatever ideology that you have, do it and have the result.

Once you have ALREADY made original 'real research' with good results, come back and complain about people doing 'implementation details'

Right now, you have no authority and no 'right' to criticize others people in public

(you are welcome do do it in private with your friends or your advisers, it's a free country )

Having said that, I would tend to agree with what you said.

[queller]

Now this may be off-topic considering this is the computer science sub-forum but in alot of schools the boundaries between computer science and computer engineering is blurred. One area this applies greatly in my research into schools for graphics and virtual reality related programs. So in my investigating of schools I have been looking at both departments.

Of course there are differences in these fields of study as far as there historical backgrounds. It appears to me that Computer Engineering in most places split from the Electrical Engineering departments with Computer Science splitting from the Math departments. However it may help you to look in both departments. With engineers caring more about getting something to work and mathematicians having a natural desire to optimize algorithms into working better.

Naturally both departments focus on both implementation details, and optimization of previous research. However maybe by expanding your search you can find something that fits your interests. However I will say both of these things in my opinion are important to be researched.

[frenzydude]

I would agree. In most universities in India like mine, the department is called "Computer Science and Engineering" .

[OH YEAH]

I don't want to be rude, but I just don't like this kind of whining in public.

Whatever they do, let them do. Whatever you do, do it.

If you can do something great in whatever ideology that you have, do it and have the result.

Once you have ALREADY made original 'real research' with good results, come back and complain about people doing 'implementation details'

Right now, you have no authority and no 'right' to criticize others people in public

(you are welcome do do it in private with your friends or your advisers, it's a free country )

Having said that, I would tend to agree with what you said.

He isn't whining, he is trying to find a school and adviser that does the kind of research that he is interested in. He isn't criticizing anyone in specific in public. He doesn't need "authority" to express his views, just as you don't need "authority" to criticize his views. I'm very pleased that this thread exists, because it is one of the few on this forum that permits actual discussion (as opposed to "yeah, I got into that school" or "no, I didn't get into that school" or "oh no, X school hasn't notified me of my acceptance yet!!") ... so please don't stifle the discussion.

Edited March 12, 2011 by OH YEAH

[OH YEAH]

I’m an undergraduate in computer science at University of Illinois. I am very passionate about answering unsolved research questions, and I’m am easily exasperated with wheel-reinventions that claim to be research.

I’ve been fortunate to find good advisors who have helped me to develop and contribute a few theoretical approaches that apply to a broad variety of applications. However, a huge volume of computer science journal/conference papers solely discuss implementation details and contribute little that generalizes beyond the specific application.

If much of the CS research community is focused on implementation and if I have to fight to find “real research questions” to study, is this a warning sign? Should I go be a mathematician or physicist, or am I not seeing the big picture of CS research?

If I stay in CS, what strategies can I use for finding a dissertation advisor (ideally at Berkeley, CMU, Stanford, et al.) who will be willing to focus on the big picture instead of on how I can connect software A with technology B to get a performance improvement on system C?

Don't disparage implementation. There are many advantages to implementation: for instance, 1) it admits compelling evidence that your ideas actually work (or show YOU why they don't work...) and 2) it can serve as a platform for future research (do you really want to wait for industry to build X so you can continue your research?). Implementation exposes a lot of holes. Make use of it.

That being said, when writing a paper about an implementation you should discuss the general implications and big picture as opposed to trivial engineering decisions. Regardless of who you work with, you can make a conscious decision to write good papers. Nobody in the top conferences wants to read about boring implementation details, and advisers at good schools want to publish at these conferences, so you won't be met with resistance.

In other words: don't worry about it so much, just find an adviser who is working on stuff you find exciting. If you don't like their papers, keep in mind that whatever graduate student wrote them isn't going to write your papers.

[DezB88]

He isn't whining, he is trying to find a school and adviser that does the kind of research that he is interested in. He isn't criticizing anyone in specific in public. He doesn't need "authority" to express his views, just as you don't need "authority" to criticize his views. I'm very pleased that this thread exists, because it is one of the few on this forum that permits actual discussion (as opposed to "yeah, I got into that school" or "no, I didn't get into that school" or "oh no, X school hasn't notified me of my acceptance yet!!") ... so please don't stifle the discussion.

please help me if I misunderstand these

"I’ve been fortunate to find good advisors who have helped me to develop and contribute a few theoretical approaches that apply to a broad variety of applications"

and

"If much of the CS research community is focused on implementation and if I have to fight to find “real research questions” to study, is this a warning sign?"

the way I see it:

only "theoretical approaches that apply to a broad variety of applications" are "real research" is what seems to be implied here.

I also don't think that people " have to fight to find “real research questions” to study"

I am not in system are, but I am sure there are a lot of real research questions out there, saying you having difficulty 'finding real research questions' is what I don't agree at all.

[gradSchoolBound]

I think @Zep has a valid point. I don't see how this is whining in public. He is making observation about general research trend in systems. What he expresses here has been acknowledged by experts in this field. He is not calling names either. This is a public forum meant to discuss ideas like this. Hence, I don't see any reason why anyone should be offended. Through examples and reasoning one could persuade @Zep how his beliefs are wrong if one feels the necessary to do so rather than ask him to shut up or discuss things in private. Through discussions of ideas and acknowledge of problems is how we move forward.

[timuralp]

Could this please be more specific? Look through SOSP, OSDI, NSDI. What is an example of "real research" vs "implementation"? Personally, I try to think of interesting ideas, think of how to create the system around them (design), consider any *interesting* implementation challenges, and evaluate the system (evaluation). The benefit of building the whole thing is that in the process of running experiments with it, one might learn something unknown about the world in which it is deployed. Maybe it is affected by a feature of a certain other system in a weird way that no one would think of otherwise.

[the poisoned pawn]

A computer scientist can be as mathematical and theoretical as any other field. One of my CS professor who is doing a lot of theory in machine learning had a PhD in Math (in one of the top 5 school for Math), but his PhD advisor is a computer science guy with CS degree.

I totally lose my interest in system due to the bad grades and the awful professors who taught my OS, architecture courses. However, I saw a lot of professors with mathematical background do system research at my school. Either I'm missing something or ...

Edited March 12, 2011 by the poisoned pawn

[symbolic]

What I like about AI in particular is that, because the software and systems that you build are so complex and detailed (often extending others' systems), with so many components, researchers have often given up largely describing the implementation in their published papers--they give a quick overview, if that, and instead rely on your ability to go to their website (or their group's, or the project's) and find out the implementation details yourself. The rest focuses on the conceptual impetus for the paper, the theory, the algorithms, etc.

I think that given the advent of the internet, CS is one of the disciplines uniquely positioned to split its academic work in two: the actual papers discuss more of the ideas behind the research, and the rest is left up for you to discover online. I don't know whether this is actually relevant to what you mean, since I'm not much of a systems person, but I can understand the sentiment.

Edited March 14, 2011 by symbolic

[zep]

Thanks for your help, everyone!

Don't disparage implementation. There are many advantages to implementation: for instance, 1) it admits compelling evidence that your ideas actually work (or show YOU why they don't work...) and 2) it can serve as a platform for future research (do you really want to wait for industry to build X so you can continue your research?). Implementation exposes a lot of holes. Make use of it.

I agree that implementation is important. I think that industry software developers are traditionally more implementation focused than researchers are. Sometimes it's hard to determine the point a researcher should say "We've fleshed out the research-y part of this idea. Let's publish it and see if Microsoft implements a version of our idea and launches it in a future product."

In other words: don't worry about it so much, just find an adviser who is working on stuff you find exciting. If you don't like their papers, keep in mind that whatever graduate student wrote them isn't going to write your papers.

Good point!