I think my original question was perhaps a bit misguided and I will need to revise the scope of what I'm asking, but I will share what I have learned.
First, Peter Shor's answer was able to guide me to some publications by Shannon which opened up a number of different papers to pursue. I wouldn't have found this information without his reply, so I am grateful for the response.
From the sources that I found, there is never any hint of skepticism about classical computation. What does happen to be a concern, however, is the reliability of complex systems built from unreliable components. Knowing what I do now, I think the question closest to my original inquiry that I am able to answer is something like
Knowing that early electronic components were unreliable, how is it possible to build a machine from unreliable components that happens to work?
I think this is a rather intuitive claim, that a complicated system made of unreliable components seems like it would be in a faulty state more often than a simple system, and as the system becomes more complicated, it should become less reliable (and I think this is what Scott Aaronson was getting at).
Some literature of the time making this claim, Smith in  says
However, the problem of making complex electronic equipment reliable still remains. Statistically, the greater the number of parts, the greater the chance of failure.
And Mine says in 
During the past several years, the electronic equipment has tended rapidly towards increased size and complexity. The more complex and enormous an equipment is, the more likely it is that any one of its elements will fail. Thus it is unavoidable that electronic equipment become unreliable.
The lecture series from von Neumann  is some of the earliest literature addressing how to make a reliable system from unreliable components, though Mullin  says the paper is an extension of earlier work by McCulloch and Pitts .
In section 10 of the paper, von Neumann says
Section 10 is devoted to a sketch of the statistical analysis necessary to show that, by using large enough bundles of lines, any desired degree of accuracy (i.e. as small a probability of malfunction of the ultimate output of the network as desired) can be obtained with a multiplexed automaton.
Despite this possibility, it seems that electronic equipment of the time had rather poor performance characteristics in practice. For example, Carhart says in 
Remarkable technological advances have been made in the field of electronics during the past decade. The impact of these developments on military activities has been dramatic and far-reaching. During World War II, radar, the proximity fuze [sic], fire-control equipment, and sonar played crucial roles in winning the victory. Since V-Day, the importance of electronics in warfare has greatly increased. It has brought about revolutionary changes in the rapidity and range of communication, in the speed and precision of controlling modern weapons, and in the detection and tracking of enemy weapons. To realize these potentialities it is imperative that electronic equipment operate reliably in the field. Yet it is generally agreed that present electronic equipment is unreliable to a serious degree. [emphasis original] For example, a recent widely distributed report (Ref. 2) states that in 1950 only about one third of Navy electronic equipment was operating properly.
Interestingly, in the final pages of the report Carhart claims that military equipment is approaching a critical complexity limit in terms of reliability, at least for serial systems. This is the only instance I found of anyone claiming a limit to extant complexity, though I think this is a rather soft claim, as the author proposes redundancy and parallel solutions to increase reliability.
I found a small number of other papers that casually make claims against complexity and then refute these arguments. For instance, Lipp in  says
Although it is commonly believed that a circuit containing a multiplicity of
elements is less reliable than a simpler one, this is untrue in a large class of
Moskowitz and McLean say in 
Hence, an increase in complexity usually leads, with present design concepts, to a decrease in equipment reliability. Yet reliable operation is as important as the satisfactory electronic solution of the other aspects of the problem.
Can some principle or concept be found in equipment design which will not tend to increase the probability of failure with increase in complexity? It is the purpose of this report to indicate a possible solution.
I haven't found any evidence of skepticism about classical computing. Instead, I did find a number of papers addressing concerns about system reliability, stemming from the original research done by von Neumann. The report by Carhart was the only instance of a claimed limitation on system reliability, but the author proposes solutions to address the problem.
Addendum ~21 months after initial post. I finally got a copy of the NEC 1954 conference proceedings. There's an article  that references the Rand Corporation report (), which says the following:
Present day electronic equipment exhibits a degree of reliability far below what is needed or desired. In 1950, for example, it was reported that only about one-third of the military electronic equipment was operating satisfactorily. On the other hand, in the past few years, designs have been radically improved to attain the greater precision and higher speeds of operation essential to modern military situations. The penalty for such an improvement in performance, with its attendant complexity, has been a corresponding increase in the difficulty of maintaining reliability. We are faced then with the anomaly that, as we design our equipment to perform more and more astonishing feats of precision and complexity, we have less and less confidence that it will perform at all.
That's the only reference in the paper that mentions some kind of limitation.
 W. S. McCulloch and W. Pitts, "A logical calculus of the ideas immanent in nervous activity," The Bulletin of Mathematical Biophysics, vol. 5, no. 4, pp. 115–133, Dec. 1943.
 A. C. Block, "A Redundancy Analog," IRE Transactions on Reliability and Quality Control, vol. PGRQC-12, pp. 1–7, Nov. 1957.
 R. R. Carhart, "A Survey of the Current Status of the Electronic Reliability Problem", white paper from RAND Corporation, 1953.
 Z. W. Birnbaum, J. D. Esary, and S. C. Saunders, "Multi-Component Systems and Structures and Their Reliability," Technometrics, vol. 3, no. 1, pp. 55–77, Feb. 1961.
 R. Gordon, "Optimum Component Redundancy for Maximum System Reliability." Operations Research, vol. 5, no. 2, pp. 229–243, 1957.
 J. von Neumann, "Lectures on Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components," Jan 1952.
 H. Mine, "Reliability of a Physical System," IRE Transactions on Circuit Theory, vol. 6, no. 5, pp. 138–151, 1959.
 L. Hellerman and M. P. Racite, "Reliabllity Techniques for Electronic Circuit Design," IRE Transactions on Reliability and Quality Control, vol. PGRQC-14, no. 0, pp. 9–16, Sep. 1958.
 J. C. Hudson and K. C. Kapur, "Reliability theory for multistate systems with multistate components," Microelectronics Reliability, vol. 22, no. 1, pp. 1–7, Jan. 1982.
 E. F. Moore and C. E. Shannon, "Reliable circuits using less reliable relays," Journal of the Franklin Institute, vol. 262, no. 3, pp. 191–208, Sep. 1956.
 A. A. Mullin, "Reliable stochastic sequential switching circuits," Transactions of the American Institute of Electrical Engineers, Part I: Communication and Electronics, vol. 77, no. 5, pp. 606–611, 1958.
 F. Moskowitz and J. B. McLean, "Some reliability aspects of systems design," IRE Transactions on Reliability and Quality Control, vol. PGRQC-8, pp. 7–35, Sep. 1956.
 T. A. Smith, "The background of reliability," IRE Transactions on Reliability and Quality Control, vol. PGRQC-8, pp. 55–58, Sep. 1956.
 J. P. Lipp, "Topology of Switching Elements vs. Reliability," IRE Transactions on Reliability and Quality Control, vol. PGRQC-10, no. 0, pp. 21–33, Jun. 1957.
 A. H. Benner and B. Meredith, "Designing Reliability into Electronic Circuit," from Proceedings of the National Electronics Conference Vol. X 1954, pp 137. Feb 1955.