Men or machines?
Recent discussions on the introduction of advanced data science in clinical practice and its impact on the physician’s role are situated in a wider discursive context which considers digitisation as a key aspect in the development of contemporary societies. The ML applications are not only increasingly present in health care but also in various other branches, such as management, jurisprudence, journalism and architecture. Advanced computer technology in many knowledge-intensive sectors has already helped to deal with complex tasks which had previously typically been the subject of human expertise. Academic jobs are, thus, not exempt from digitisation but today stand right at the centre of being supported (or partly replaced) by systems of so-called AI. Whereas techno-optimist positions welcome the increasing usage of computational systems at the workplace [17, 18], more sceptical authors warn of an ‘AI takeover’ which could challenge human expertise and competence .
The awakening interest of the medical profession in ML_CDSS has been raised, not least due to the emergence of studies comparing physicians’ clinical skills with the performance of ML-based applications. The—mainly visual—competence of a dermatologist in classifying skin lesions, for example, has been shown to be comparable with the respective performance of an automated classification system running with deep convolutional neural networks. A deep convolutional neural network was trained with about 130,000 clinical images of skin lesions in a comparative trial . The model’s performance was then tested against 21 board-certified dermatologists in classifying images of benign versus malign lesions. Consequently, the deep convolutional neural network outperformed most of the clinicians regarding the sensitivity and specificity of the diagnoses. Other studies from dermatology operating with similar methods show that convolutional neural networks surpass clinicians’ diagnostic performance particularly regarding the specificity of the results . Similar promising findings for the use of ML in diagnostics have been derived concerning chest X-ray evaluation  and arrhythmia detection in electrocardiograms . More complicated diagnostic tasks have also been the subject of trials comparing the performance of clinicians and algorithms. Liang et al., for example, applied ML classifiers to about 1.3 million electronic health records from paediatric patients . An automated natural language processing system applied deep learning techniques to extract clinically relevant information and was trained to imitate physicians’ hypothetico-deductive reasoning. The model’s performance was compared to the diagnoses of 20 paediatricians of varying professional experience. Consequently, the average score achieved by the model was higher than in the junior physician groups but lower than in the senior physician groups.
Methodological details (e.g. the sample size, composition of the control group or ecological validity) of such comparative studies demand further scrutiny and the results cannot be easily generalised to other diagnostic branches or clinical practice so far. The discussion on the ethical implications of integrating ML_CDSS at the physician’s workplace, however, is already in full swing [11, 25,26,27]. Similar to digitisation in general, positions regarding the future of the medical profession in digitised health care are markedly divided to date: optimist positions highlight the potential of advanced data science for promoting a personalised, patient-centred or even more humane patient care . According to this view, technical support systems, for example, could be applied for creating better options for physicians to invest in relational and empathetic aspects of their practice. By contrast, other authors stress that ML systems have not yet proven their positive effects on health outcomes and may cause unwanted effects on professional practice, such as an over-reliance on computers, deskilling or a loss of confidence in providing diagnoses .
Against this background, it is not surprising that physicians’ institutional bodies have tackled the issue of digital decision support intensively in scientific conferences and events, statements and opinion papers [30, 31]. In addition, reference to the ‘profession’ is widespread in the current discourse on the digitisation of health care , whereas a more elaborate way of dealing with medical professionalism and professionalisation theory has so far been missing. This article, therefore, takes up the thread of the current discourse on the ethical implications of ML_CDSS and aims to add a systematic analysis through the lens of professionalisation theory.
Professions and professional action have been the subject of sociological investigation for almost a century (e.g. [32, 33]). Many scholars have been concerned with the institutional and organisational characteristics and dynamics of professions on a macro level (e.g. [34,35,36,37]). By contrast, others, particularly the German-speaking sociology of professions (building on classical works such as [32, 33, 38, 39]), have focused on the micro level, thus, the structural features of professional action in general. Regardless of the various theoretical approaches, it has been widely recognised that, in simple terms, professional action generally deals with practical problems or crises of people (clients or patients) and usually draws on scientific knowledge. However, it cannot simply apply this expertise in terms of schematic rules because it is confronted with contradictory requirements and uncertainties of various kinds.
In order to avoid misunderstandings, it has to be stressed that the counterpart of the role of the professional has been called a ‘client’ since the beginnings of the sociology of professions . The client is the person who engages professional advice or service. In common parlance, this term is used, for instance, regarding the role pair of lawyer and client. A client in this sense is neither a customer nor a consumer, both known from economic contexts. Thus, wherever we speak of a client as the counterpart of the medical professional in this article, we mean nothing else than the classic role of the patient. Furthermore, as will become clear below, from the perspective of professionalisation theory, a reduction of the role of the patient to something like a customer or consumer (as it has been empirically observed and ethically criticised in the context of an economisation of health care) would be a clear sign of deprofessionalisation.
The ideal type of client-related professionalised action has been specified and tested in numerous empirical studies of medical practice in the professionalisation theory developed by Oevermann [40, 41]. According to this elaborated approach, a professionalised medical practice is characterised by five structural features :
The patient has a concern resulting from a problem or crisis that affects her or his everyday life and that she or he cannot solve alone or autonomously. Therefore, the patient seeks help from a professional. Although this seems self-evident, not being in need of help or advice is a bad prerequisite for professional action and may cause difficulties.
The task of the physician is to cope with or solve the problem or crisis of the patient vicariously. Thus, the physician deals with the patient’s concern or problem. Again, this seems self-evident, but it is important that the physician works, in fact, on the patient’s concern, whatever it might be, and not on something else that the professional might consider to be important or the ‘real’ issue.
The physician deals with the patient’s concern together with the patient. The patient can neither simply leave her or his problem (or sick body) at the counter like a defective device, nor can the physician solve the problem without the patient. Therefore, both physician and patient must enter into a working alliance in order to tackle the problem in focus together. By doing so, the physician is confronted with a general tension that she or he has to withstand rather than resolve unilaterally. That is the tension between the patient’s autonomy and dependency vis-à-vis the professional. On the one hand, the patient might place her/himself in a particularly pronounced dependency, for instance, on the operating physician. On the other hand, the patient has to autonomously consent to this relinquishment of autonomy. Therefore, the physician gives assistance without patronising.
In addition, the physician regards the patient in a holistic manner without building up a private relationship. This concerns another tension, namely, between the diffusivity and role-relatedness of the physician–patient relationship. On the one hand, the patient approaches the doctor as a ‘whole person’, thus, not bound to a specific role but rather diffuse, as in family or friendly relationships. Indeed, everything about the patient and her or his history can be interesting or necessary for the physician, even intimate details from the patient’s private life. On the other hand, this diffusivity also has its limits. The patient always remains relevant in the role of a service recipient. Thus, the physician is interested in the patient’s personal matters but only insofar as they are relevant to the physician’s professional task of providing medical assistance and not out of curiosity or personal interest.
Finally, the physician applies her or his general expertise to the particularities of the individual case. This addresses a third tension, namely, between the knowledge base and case specificity. An individual case is always a ‘case of X’, thus, a representation of a general structure or relationship . It is the task of the professional to find an understanding of the patient’s problem which is, on the one hand, adequate to her or his general medico-scientific knowledge and gives, on the other hand, expression to the particularities of the patient’s individual problem situation.
Due to the three tensions outlined above (autonomy vs. dependency, diffusivity vs. role-relatedness and knowledge base vs. case specificity), professionalisation theory does not consider professional medical action to be essentially standardizable. This does not mean that there are or should not be standards of ‘good medical practice’ and corresponding guidelines. However, these cannot be applied in a schematic, formalistic way. Conversely, the key characteristic of professionalised action is to withstand the three tensions. Resolving them unilaterally would mean to act in a non-professionalised manner.
Ethics of ML_CDSS from the perspective of professionalisation theory
In this section, we take the five structural features of professionalised medical practice introduced above as an analytical lens for gaining a closer look at the use of ML_CDSS in medical practice. We reflect on the five structural features separately, in each case integrating the perspective of professionalisation theory with a discussion of ethical implications. We presuppose in our analysis that the ML_CDSS is integrated into the physician’s workflow and does not interact directly with the patient. Patient-side ML applications (such as symptom checkers or fitness apps) are, thus, not considered even if they are similarly associated with important ethical issues in light of professionalisation theory.
The patient has a concern. This also holds true, of course, in a medical practice integrating ML_CDSS. However, it might be less clear, particularly in practices dealing with prevention, whether or to what extent a patient has a concern at all. Here, an increasing routinization, most notably in highly specialised clinical institutions, of clarifying whether specific risk dispositions are existent or not runs the danger of ignoring the question to what extent such a clarification is really in the patient’s interest. The general availability of an ML_CDSS may, thus, enforce a physician’s attitude to routinely feed the ML_CDSS with the patient data at hand without engaging in building up a sound and professional physician–patient relationship by asking, for instance, the initial question “What is your concern?” or “How can I help you?”.
These initial and relationship-building questions are also crucial for scrutinizing whether the special expertise of the physician is appropriate to tackle the patient’s concern and, more generally, whether it is actually a medical issue that motivates the patient to seek the physician’s help and not a problem originating, for example, solely from the job or private situation. The availability of technological resources and the increasing inclusion of information other than biomedical data (e.g. data on mobility, nutrition or from social media) increasingly blurs the border between the medical sphere and ‘private life’ and might prompt the physician to deal with issues that do not lie within the scope of her or his competency. If the patient’s concern does not fit the physician’s field of expertise, it is the task of the latter to point out this incongruity and, ideally, refer the patient to another, more appropriate professional field.
The physician deals with the patient’s concern. Taking this seriously, it means that the physician uses the ML_CDSS only and exclusively if this is motivated by the patient’s concern. At least two questions arise here: are other (legitimate) interests influencing the physician’s use of the ML_CDSS and do additional issues arise from the use of the ML_CDSS which go beyond the patient’s initial interest?
Regarding the first question, an ethically reflective employment of ML_CDSS should consider that the enormous technological options inherent in these systems might lead to forms of employment which deviate from dealing with the patient’s concern in a strict sense. An obvious field of controversy lies in the intersection between clinical practice and research. It has already been highlighted that the borders between these two fields become increasingly unclear in the use of ML_CDSS . As the machines continue to develop in their use (‘learning’), patients are typically feeding their data and contribute to a training and refinement of the algorithmic tools. Whereas it can be considered as an ethical imperative to continuously develop the quality of learning health-care systems , the blurred boundaries between clinical care and research might place high requirements on patient information and could prevent physicians from taking measures which are directed exclusively to the individual patient’s welfare. Well-established guidelines exist on how to deal with conflicts of interest, for example, in clinical studies or guideline development . The situation seems to be less clear, however, regarding potential institutional directives or economic incentives to use or test an ML_CDSS in practice and let it develop further based on patient data.
The other question concerns, for example, the occurrence of secondary findings which were not targeted by the diagnostics and, thus, do not necessarily mirror the patient’s concern. Secondary findings arising from ML_CDSS might extend, for instance, to risk predictions of various adverse events (such as death or cardiovascular complications) or findings about the patient’s supposed lifestyle or therapeutic adherence. From the perspective of professionalisation theory and an ethical perspective, the eventuality of secondary findings and how the physician is supposed to deal with them should be discussed and reflected on together with the patient. Only if the patient includes such further findings (at best specified) in her or his concern before the use of the ML_CDSS does the physician receive the mandate to later reveal and share them with the patient.
While dealing with the patient’s concern, the physician gives assistance without patronising (tension: autonomy vs. dependency). Regarding ML_CDSS, there are several important aspects to discuss. Firstly, if the use of an ML_CDSS implies any risks to the diagnostic or therapeutic process, these risks need to be made transparent and discussed with the patient. Such risks may include a high rate of false positive or false negative results, or problematic secondary findings, as discussed above. In the case of truly severe risks associated with ML_CDSS and approved available alternative procedures, such alternatives should be offered, discussed and made accessible to the patient. Particularly regarding secondary findings, it could appear to be helpful to learn from other fields, such as medical genetics and genetic counselling, a highly professionalised practice  where the patient’s ‘right not to know’ has been intensively discussed and thoroughly implemented . Unilaterally resolving the tension between the patient’s autonomy and dependency (here, towards dependency) would mean concealing any risk or other information that might restrain the patient from consenting or adhering to the use of an ML_CDSS.
Another important aspect concerns, again, the degree of routinization of the ML_CDSS. The more routinized or automatically (in the sense of unquestioned) the physician follows the output or recommendations of an ML_CDSS, the higher the patient’s dependency on this ML_CDSS. Therefore, the physician needs to compensate this ML_CDSS-dependency by strengthening the patient’s autonomy. This might be realised, for example, by explaining the functionality of the ML_CDSS and potential shortcomings, pointing to alternative interpretations that complement the ML_CDSS output and supporting the patient in finding a decision regarding all reasonable options for further treatment, if applicable. It has to be noted, however, that attempts to increase the patients’ digital health literacy  might remain limited in light of the complexity and opacity of advanced ML technologies. Patient education is, thus, closely related here to technological challenges of explainability in advanced data science . Nevertheless, unilaterally resolving the tension between the patient’s autonomy and dependency (here again, towards dependency) would mean to follow any recommendation of an ML_CDSS in a unreflected way and, simultaneously, to refrain from problematising the ML_CDSS output vis-à-vis the patient, even if there are good reasons to do so.
Such an impact of ML_CDSS on patient autonomy can also be discussed in the light of a ‘computer-paternalism’ (“Computer knows best”), which might become apparent when AI systems recommend and rank treatment options without considering the individual patient’s values and preferences . The ML_CDSS might then be depicted as exhibiting a patronising ‘attitude’ towards the patient. Bioethicists, therefore, plead for a value-flexible design of AI systems  and a meaningful and reflective integration of these supporting systems into the physician–patient relationship .
Further issues in the intersection between ML_CDSS, patient autonomy and dependency include the question whether patients should have the right to refrain from the use of AI systems and, instead, to stick to the less advanced alternatives. On the one hand, it can be sensibly argued that patients should be able to withdraw from AI diagnostics and treatment due to the specific role of the physician, bias and opacity problems, or the future impact of AI systems on the health-care system . On the other hand, health-care systems could rapidly become strained if they need to hold available both alternatives in the future: medical technologies based on advanced data science and their ‘analogous equivalents’. If ML_CDSS increasingly prove to be effective and secure, there need to be good reasons to grant ‘opt-out’ rights to patients as there is generally no patient right to stick to outdated procedures in health care. In addition, from the point of view of professionalisation theory, preserving the patient’s autonomy means to enable her or him to choose between alternative, justifiable options. However, leaving the decision to the patient alone without providing information, giving advice or supporting her/him in finding the individual ‘right’ decision would similarly resolve the tension between the patient’s autonomy and dependency in a unilateral way (but this time, towards autonomy; the term ‘autonomy’ here is not used in the bioethical sense but from the viewpoint of professionalisation theory; it could, in this case, be translated to ‘independency’).
The physician regards the patient in a holistic manner without building up a private relationship (tension: diffusivity vs. role-relatedness). Using an ML_CDSS in medical practice entails the risk of ignoring patient-related information and aspects that are, in fact, relevant from a holistic medical perspective but cannot be fed as data into the ML_CDSS. To take a rather simple example, this may concern the right timing of an invasive intervention in relation to the general life situation of the patient, such as the current employment or family situation. Thus, the greater the importance of the ML_CDSS within the professional decision-making process, the more the physician needs to make sure that all aspects of the patient’s life situation that might be relevant for coping with the health problem and for a good health outcome are laid on the table and considered appropriately. Not taking such aspects into account would unilaterally resolve the tension between the diffusivity and role-relatedness of the physician–patient relationship (here, towards role-relatedness), as the patient is reduced to the role of a mere data provider.
On the other hand, even if the attempt to treat the patient holistically also causes a physician to consider ‘non-biomedical’ facts (such as patient’s life situation, outward appearance or behavior during the conversation), she or he must consistently stick to the professional role and should not unduly (i.e. without a good reason) intrude into the patient’s privacy. With the increasing use of ML_CDSS, the tension between diffusivity and role-relatedness is intensifying even more due to several factors: the widespread deployment of ML technologies for preventive purposes leads to a situation where the clinician might get insights into aspects of the patient’s life (e.g. mobility, nutrition, sleep habits) which could—or could not—be relevant to medical practice. The increasing use of data from non-medical contexts (e.g. social media data for psychiatric diagnostics ) leads to an extension of the medical sphere to fields in life which have previously had a non-medical character. Significant chances and challenges (e.g. emerging from the expansion of the concept of disease ), therefore, may arise from the use of ML_CDSS regarding the aim of treating patients in a holistic manner. Unilaterally resolving the tension between the diffusivity and role-relatedness of the physician–patient relationship (here, towards diffusivity) would mean to make use of such non-medical data for purposes that do no fall within the physician’s scope of responsibility.
The physician applies her or his general expertise to the particularities of the individual case (tension: knowledge base vs. case specificity). This aspect once again underlines that the physician has to ensure that information on the individual patient that cannot technically be included in the ML_CDSS is taken into account within the professional decision-making process. The more important and routinized the use of an ML_CDSS is, the more the physician needs to scrutinize whether the ML_CDSS output is still valid regarding the individual case, including all relevant information available. Ignoring such case-specific information would unilaterally resolve the tension between the knowledge base and case specificity (here, towards the general but limited knowledge base of the ML_CDSS).
In addition, this tension points to the matter of trust (in the ML_CDSS), an issue widely discussed in the literature on ML and AI in general, including trustworthy or explainable AI [54, 55]. The physician needs to be able to trust (and periodically reaffirm) that the ML_CDSS represents (or outperforms) her or his own medico-scientific knowledge adequately and that its use improves the outcome of her or his medical practice. The more this is uncertain, the less the physician should rely on the ML_CDSS. Still relying on the technical system, despite existing uncertainty or doubt, would again unilaterally resolve the tension between the knowledge base and case specificity (here again, towards the knowledge base of the ML_CDSS). The reverse case, resolving the tension towards case specificity, would mean to get lost in the particularities of the individual case and neglect to relate them in a meaningful way to the general knowledge base of the ML_CDSS.
The question to what extent ML_CDSS are equipped for dealing with individual cases, finally leads to a particularly challenging aspect in the ethical evaluation of such systems. As outlined above, recent studies indicate that AI systems meet or even surpass physicians’ competencies regarding specific, isolated tasks. This punctual superiority is likely to grow in the near future as the technological advancement progresses. However, it becomes increasingly obvious that automated procedures of ML (that is part of the knowledge base from the perspective of professionalisation theory) can promote biases and disadvantage certain groups of patients who have not, for example, been adequately represented in the training data. In addition, based on the training data, algorithmic biases may reproduce, for instance, racial biases already present in health-care practice [56, 57]. The application of ML in health care, therefore, needs permanent surveillance because incorporating a particular practice into an algorithm may imply an unsubstantiated legitimacy of that practice which is not justified by an improved outcome for the individual patient . Whereas ML_CDSS might give rise to the impression that subjective human judgement is replaced with unadulterated data-driven recommendations, in fact, ‘new’ forms of discrimination can arise within data technologies, for example, when labelling or annotating the data. Two aspects need to be considered when using a big data knowledge base for the care of individual patients: the risk of perpetuating wrongful clinical practices inherent in the training data and illegitimate discrimination arising from the algorithmic nature of the decision support itself.