header advert
Bone & Joint Open Logo

Receive monthly Table of Contents alerts from Bone & Joint Open

Comprehensive article alerts can be set up and managed through your account settings

View my account settings

Visit Bone & Joint Open at:

X (Twitter)
Article alerts Social media
Current issue
Submit
Open Access

Arthroplasty

Arthroplasty information on the internet

quality or quantity?

Download PDF

Abstract

Aims

Total joint replacement (TJR) is a high-cost, high-volume procedure that impacts patients’ quality of life. Informed decisions are important for patients facing TJR. The quality of information provided by websites regarding TJR is highly variable. We aimed to measure the quality of TJR information online.

Methods

We identified 10,800 websites using 18 TJR-related keywords (conditions and procedures) across the Australian, French, German and Spanish Google search engines. We used the Health on the Net (HON) toolbar to evaluate the first 150 websites downloaded for every keyword in each language. The quality of information on websites was inspected, accounting for differences by language and tertiles. We also undertook an analysis of English websites to explore types of website providers.

Results

‘Total joint replacement’ had the most results returned (150 million websites), and 9% of websites are HON-accredited. Differences in information quality were seen across search terms (p < 0.001) and tertiles (p < 0.001), but not between languages (p = 0.226). A larger proportion of HON-accredited websites were seen from keywords in the condition and arthroplasty categories. The first tertile contained the highest number of HON-accredited websites for the majority of search terms. Government/educational bodies sponsored the majority of websites.

Conclusion

Clinicians must consider the shortage of websites providing validated information, with disparities in both number and quality of websites for TJR conditions and procedures. As such, the challenge for clinicians is to lead the design of reliable, accurate and ethical orthopaedic websites online and direct patients to them. This stands to reward both parties greatly.

Introduction

Patients use the internet for clarifying their understanding of diagnoses.1,2 Over 80% of patients, clinicians, allied health, and invested third parties do this because of the belief that the internet is a reliable, trustworthy and accessible source,3,4 and up to 35% of patients use it to self-diagnose without clinician follow-up.5 In addition, over 70% of adult consumers sought health information on the internet in recent years, and this is predicted to rise.6

There is an extensive and unregulated range of medical and procedural information on the internet that can potentially impact peoples’ expectations and decision-making.7,8 Amid an overwhelming amount of information, the internet can be misleading if patients lack health and e-health literacy skills to find accurate and relevant information.9-11 This can be a difficult skillset to acquire for particular subsets of patients, such as non-English speaking patients and the older demographic, who, despite showing increasing internet usage, may lack awareness and general know-how of using technology.7,12,13

Language is known to affect the quality of information14-18 and this may impact the reliability of information that is available to patients living in diverse communities or without English-speaking backgrounds. Therefore, assessment of the reliability of information on the internet is merited.

Healthcare professionals, institutions, and industry groups develop websites for commercial advantage.3 In contrast, only a small percentage of websites originate from government/educational and non-profit organizations, raising concerns about objectivity, bias, and accuracy of information sources.19 There is a need for greater education and easy-to-use tools that can assist patients and clinicians in ascertaining quality information online.16,17

Total joint replacement (TJR) is a common and proven intervention for many patients with end-stage osteoarthritis, and is one of the highest volume medical procedures worldwide.20 To date, no studies have evaluated online TJR information. Our aim was to quantify the quality of online TJR-related information across several common Western languages, and categorize information by website sponsor.

Methods

Our methodology has previously been described.16,17,21 Reliable health information on the internet can be found using a number of online instruments.22-25 A consideration of the key characteristics of each of these tools is presented in Table I.15,16,26 To analyze a large number of websites, we chose to use the Health on the Net (HON; Chêne-Bourg, Switzerland) tool for its practicality and time-efficiency compared to other tools, which require manual input and tabulation of results. HON is a not-for-profit multilingual (34 languages) accreditation entity endorsed by the World Health Organization. It certifies health websites according to eight key HON principles:15 authority, complementarity, privacy, attribution, justifiability, transparency, financial disclosure, and advertising policy. Website owners can apply for HON accreditation, after which an international, independent, qualified accrediting body of the HON team checks the website for any deficiencies of the HON principles that need to be refined, ensuring that HON accreditation meets high international benchmarking.

Table I.

Evaluation of instruments used to assess quality of information on internet.

Instrument Details Advantages Disadvantages
HONcode Internet toolbar function that determines HON accreditation status Free to download and use; accessible to patients and clinicians; WHO support Voluntary subscription fee for websites to become HON-accredited; good quality sites may lack accreditation and not be trusted
DISCERN22 Assesses the quality of information regarding treatment choices online Free to use Time-consuming, complex, requires manual and subjective assessment of components23
LIDA24 Assesses website design and content across accessibility, usability and reliability Free to use Time-consuming, complex, requires manual and subjective assessment of components23
United States Department of Health and Human Services, ODPHP National Quality Health Website Survey25 Measures the reliability and usability of website information Broad coverage of website attributes including website sponsor Subjective and laborious for lay users23,25

  1. HON, Health on the Net; WHO, World Health Organization; ODPHP, Office of Disease Prevention and Health Promotion.

We installed the HONcode toolbar, an algorithm encompassing the HON principles that determines if a website is HON accredited or not.15 This toolbar automatically activates as a visual cue for users if a website has HON accreditation. The toolbar is free and simple to install on any personal computer and can be utilized by patients and clinicians to assess website quality.16 The HONcode function is a high-calibre instrument as determined by several studies.16,17 It offers excellent overall direction for users to assess the reliability and objectivity of a website.27,28 Importantly, a number of studies have utilized the HONcode tool to evaluate website quality, with between 7% and 27% of websites accredited.19,29-31

In this study, we used the Australian, French, German, and Spanish Google search engines for each respective language search. A free-to-use algorithm32 was constructed and coded that automatically searched Google by inputting search term, language, and number of items to be returned. This algorithm was able to determine whether a website is HON accredited. A Microsoft Excel spreadsheet with this information was subsequently generated and the data mined for relevant information. Using our constructed algorithm, we performed an internet search of 18 terms between April and September 2019 (Table II) and assessed 10,800 websites for HON accreditation. English medical language has been used for search terms across each of the other languages on their respective Google search engines. The terms searched were: arthritis; osteoarthritis; end-stage arthritis; bone on bone arthritis; total joint replacement; total joint arthroplasty; total hip replacement; total knee replacement; total hip arthroplasty; total knee arthroplasty; anterior hip replacement; posterior hip replacement; unicompartmental knee replacement; half knee replacement; Mako knee; Oxford knee replacement; total hip replacement recovery time; and total knee replacement recovery time. These terms were selected as the most common, relevant, and topical TJR-related terms and procedures used to search for information on the internet. These terms were informed by expert surgeon input (PFC), scanning online patient forums and surveying 15 patients who presented for TJR consultation at a large public hospital in Australia.

Table II.

Number and percentage of HON-accredited websites.

Category/search term Total websites returned HON accredited (600 per term) p-value*
HONcode+† HONcode-‡ Total HONcode, %§
Condition
Arthritis 109,000,000 97 503 600 16
Osteoarthritis 16,900,000 121 479 600 20
End-stage arthritis 43,700,000 56 544 600 9
Bone on bone arthritis 118,000,000 96 504 600 16
Total 76,350,000 370** 2,030** 2,400** 16 < 0.001
Arthroplasty
Total joint arthroplasty 6,070,000 71 519 600 12
Total joint replacement 146,000,000 81 519 600 14
Total 76,035,000 152** 1,048** 1,200** 13 0.385
Hip
Total hip arthroplasty 5,240,000 72 528 600 12
Total hip replacement 70,000,000 60 540 600 10
Total 37,620,000 132** 1,068** 1,200** 11 0.268
Knee
Total knee arthroplasty 6,460,000 79 521 600 13
Total knee replacement 110,000,000 71 529 600 12
Total 58,230,000 150** 1,050** 1,200** 13 0.485
Approach
Anterior hip replacement 9,280,000 32 568 600 5
Posterior hip replacement 6,200,000 46 554 600 8
Total 7,740,000 78** 1,122** 1,200** 7 0.101
Unicompartmental knee
Half knee replacement 31,100,000 48 552 600 8
Unicompartmental knee replacement 240,000 38 562 600 6
Mako knee 2,370,000 14 571 600 2
Oxford knee replacement 117,000,000 29 571 600 5
Total 16,735,000 129** 2,271** 2400** 6 < 0.001
Recovery
Total hip replacement recovery time 15,100,000 76 524 600 13
Total knee replacement recovery time 15,200,000 77 523 600 13
Total 30,300,000 153** 1,047** 1,200** 13 0.931
Overall total 30,300,000 1,011** 8,589** 10,800** 12 < 0.001
  1. *

    Pearson chi-squared test, with < 0.05 indicating significant number of HON-accredited websites returned during a search.

  1. †

    HON-accredited website.

  1. ‡

    Not HON-accredited website.

  1. §

    Percentage of HON-accredited websites, calculated by HONcode+ divided by the total websites ((HONcode+) + (HONcode-))

  1. Median.

  1. **

    Sum.

HON-accredited website internet search

Patients seldom read websites beyond the first page of results,33 so the first 150 websites (approximately 15 pages) returned for each search term from our algorithm were screened for HON accreditation. This was to determine if any reliable online information was potentially being missed by internet users.

Tertile analysis of accredited websites

Additionally, each search term’s 150 returned websites were split into tertiles (first 50, middle 50, and last 50), as described in previous studies.19,29,34 For each tertile, the percentage of HON-accredited websites were analyzed and compared across languages by a chi-squared test. This was to determine where reliable websites appeared most frequently, namely in the pages most likely (first tertile) to least likely (third tertile).

Quality assurance

For quality control against our constructed algorithm, we manually evaluated all websites of a randomly selected control term, “arthritis”, as well as the non-accredited sites using the HON principles to determine if the sites were HON-accredited. This was first to check fidelity of our own constructed algorithm in finding HON-accredited sites against non-accredited sites. Second, by manually evaluating websites with the HON principles in mind, we could also ascertain if a website fulfilled the criteria to be HON-accredited despite not being officially accredited. Previous studies have identified that approximately 5% of websites are worthy of HON accreditation, but have not yet been accredited,15-17

HON accreditation associated variables

Search term, language, and tertile were used as major variables to conduct logistic regression. The reference groups for each variable were arthritis, English, and the first tertile.

Website sponsor analysis

Website sponsorship was determined by MTD reviewing every English website for each English search term. The sponsorship groups were: lawyers; non-profit organizations; government organizations/educational institutions; commercial; orthopaedic specialists and their professional organizations; other healthcare professionals; other (social media, forums, personal websites). Sponsorship is not equivalent to Google advertisements seen on Google results pages (found at the top or sides of searches). As per other similar analyses, these advertisements were not included.16

Statistical analysis

Search term, language, and tertile proportion comparisons were performed by chi-squared tests. We conducted two-sided statistical tests, and defined significance as p < 0.05. We used multiple logistic regression to analyze odds ratios and 95% confidence intervals for search terms having HON accreditation with arthritis, English, and the first tertile as the references (Table III). Analyses were performed by Stata v15.0 (StataCorp, College Station, Texas, USA).

Table III.

Odds ratio and 95% confidence limits.

Effect on HONcode status Odds ratio 95% confidence interval p-value*
Search term
Arthritis 1.00 (referent) N/A N/A
Osteoarthritis 1.310 0.975 to 1.759 0.073
End-stage arthritis 0.534 0.376 to 0.758 < 0.001
Bone on bone arthritis 0.988 0.726 to 1.344 0.937
Total joint arthroplasty 0.696 0.501 to 0.968 0.031
Total joint replacement 0.809 0.588 to 1.114 0.194
Total hip arthroplasty 0.707 0.509 to 0.982 0.039
Total hip replacement 0.576 0.408 to 0.813 0.002
Total knee arthroplasty 0.786 0.570 to 1.084 0.142
Total knee replacement 0.696 0.501 to 0.968 0.031
Anterior hip replacement 0.292 0.192 to 0.443 < 0.001
Posterior hip replacement 0.431 0.297 to 0.624 < 0.001
Half knee replacement 0.451 0.313 to 0.650 < 0.001
Unicompartmental knee replacement 0.351 0.236 to 0.520 < 0.001
Mako knee 0.123 0.070 to 0.220 < 0.001
Oxford knee replacement 0.263 0.171 to 0.406 < 0.001
Total hip replacement recovery time 0.752 0.543 to 1.040 0.085
Total knee replacement recovery time 0.763 0.553 to 1.055 0.102
Tertile
first tertile (0 to 50) 1.00 (referent) N/A N/A
second tertile (51 to 100) 0.320 0.276 to 0.370 < 0.001
third tertile (101 to 150) 0.144 0.119 to 0.175 < 0.001
Language
English 1.00 (referent) N/A N/A
French 1.201 1.009 to 1.428 0.039
German 1.123 0.942 to 1.338 0.195
Spanish 1.119 0.939 to 1.333 0.211
  1. *

    Logistic regression, where < 0.05 indicates likelihood of HON accreditation by search term, tertile, or language to be found in comparison to the reference.

Ethics

Quality assurance approval (092/19) was obtained at St Vincent’s Hospital Melbourne to question patients about what search terms they would use to find health information regarding their TJR.

Results

The total number of websites for each TJR-related search term was variable (Table II). ‘Total joint replacement’ returned the highest number of websites (over 146 million), followed by ‘bone on bone arthritis’ (approximately 118 million). ‘Unicompartmental knee replacement’ had the fewest websites, with only 240,000.

With an overall median of 12% (interquartile range 5%), all search terms returned a low percentage of HON-accredited websites (Table II). There were 5% or fewer HON-accredited sites for search terms ‘anterior hip replacement’, ‘Oxford knee replacement’, and ‘Mako knee’ (Table II).

HON-accredited websites were a similar proportion between languages (Table IV, Figure 1), namely French (13%), Spanish (12%), German (11%), and English (9%). The first tertile (first 50 websites) had the largest percentage of HON-accredited websites (Table V, Figure 2).

Table IV.

Percentage of HON-accredited websites by language.

English French German Spanish p-value**
Category/search term +* -† %‡ +* -† %‡ +* -† %‡ +* -† %‡
Condition
Arthritis 20 130 13 29 121 19 22 128 15 26 124 17
Osteoarthritis 25 125 17 33 117 22 32 118 21 31 119 21
End-stage arthritis 13 137 9 16 134 11 13 137 9 14 136 9
Bone on bone arthritis 26 124 17 23 127 15 24 126 16 23 127 15
Total 84§ 516§ 15 101§ 499§ 17 91§ 509§ 15 94§ 506§ 16 0.983
Arthroplasty
Total joint arthroplasty 15 135 10 20 130 13 20 130 13 16 134 11
Total joint replacement 21 129 14 21 129 14 17 133 11 22 128 15
Total 36§ 264§ 12 41§ 259§ 14 37§ 263§ 12 38§ 262§ 13 0.668
Hip
Total hip arthroplasty 18 132 12 18 132 12 18 132 12 18 132 12
Total hip replacement 10 140 7 18 132 12 16 134 11 16 134 11
Total 28§ 272§ 9 36§ 264§ 12 34§ 266§ 11 34§ 266§ 11
0.696
Knee
Total knee arthroplasty 14 136 9 22 128 15 22 128 15 21 129 14
Total knee replacement 14 136 9 21 129 14 17 133 11 19 131 13
Total 28* 272§ 9 43§ 257§ 14 39§ 261§ 13 40§ 260§ 13 0.953
Approach
Anterior hip replacement 8 142 5 9 141 6 8 142 5 7 143 5
Posterior hip replacement 10 140 7 13 137 9 12 138 8 11 139 7
Total 18§ 282§ 6 22§ 278§ 7 20§ 280§ 7 18§ 282§ 6 0.988
Unicompartmental knee
Half knee replacement 10 140 7 11 139 7 14 136 9 13 137 9
Unicompartmental knee replacement 11 139 7 9 141 6 9 141 6 9 141 6
Mako knee 5 145 3 3 147 2 4 146 3 2 148 1
Oxford knee replacement 7 143 5 8 142 5 7 143 5 7 143 5
Total 33§ 567§ 6 31§ 569§ 6 34§ 566§ 5 31§ 569§ 5 0.982
Recovery
Total hip replacement recovery time 18 132 12 19 131 13 20 130 13 19 131 13
Total knee replacement recovery time 20 130 13 19 131 13 19 131 13 19 131 13
Total 38§ 162§ 13 38§ 162§ 13 39§ 161§ 13 38§ 162§ 13 0.989
Overall total 315 § 2,335 § 9 312 § 2,288 § 13 294 § 2,306 § 11 293 § 2,307 § 12 0.226
  1. *

    HON-accredited website.

  1. †

    Not HON-accredited website.

  1. ‡

    Percentage of HON-accredited websites, calculated by HONcode+ divided by the total websites ((HONcode+) + (HONcode-)).

  1. §

    Sum.

  1. Median.

  1. **

    Pearson chi-squared test, with < 0.05 indicating significant number of HON-accredited websites returned during a search.

Fig. 1 Column graph of median percentage of HON-accredited sites for all keywords arranged according to language. Each keyword was searched on native Google search engine of respective countries.

Fig. 1

Column graph of median percentage of HON-accredited sites for all keywords arranged according to language. Each keyword was searched on native Google search engine of respective countries.

Table V.

Percentage of HON-accredited websites by tertile.

Category/search term HON-accredited websites
Tertile 1 (sites 1 to 50) Tertile 2 (sites 51 to 100) Tertile 3 (sites 101 to 150) p-value*
+† -‡ %§ +† -‡ %§ +† -‡ %§
Condition
Arthritis 56 144 28 36 163 18 5 196 3 < 0.001
Osteoarthritis 59 141 30 31 169 16 6 194 3 < 0.001
End-stage arthritis 36 164 18 11 189 6 9 191 5 < 0.001
Bone on bone arthritis 70 130 35 29 171 15 22 178 11 < 0.001
Arthroplasty
Total joint arthroplasty 43 157 22 24 176 12 4 196 2 < 0.001
Total joint replacement 55 145 28 17 183 9 9 191 5 < 0.001
Hip
Total hip arthroplasty 53 147 27 13 187 7 6 194 3 < 0.001
Total hip replacement 44 156 22 11 189 6 5 195 3 < 0.001
Knee
Total knee arthroplasty 54 146 27 14 186 7 11 189 6 < 0.001
Total knee replacement 54 146 27 10 190 5 7 193 4 < 0.001
Approach
Anterior hip replacement 21 179 11 5 195 3 6 194 3 < 0.001
Posterior hip replacement 24 176 12 18 182 9 4 196 2 0.001
Unicompartmental knee
Half knee replacement 21 179 11 5 195 3 22 178 11 0.002
Unicompartmental knee replacement 5 195 3 8 192 4 1 199 1 < 0.001
Mako knee 16 184 8 10 190 5 3 197 2 0.067
Oxford knee replacement 23 177 12 15 185 8 0 200 0 0.010
Recovery
Total hip replacement recovery time 62 138 31 13 187 7 1 199 1 < 0.001
Total knee replacement recovery time 56 144 28 10 190 5 11 189 6 < 0.001
Overall total 752 2,848 25** 280 3,319 7 ** 132 3,469 3 ** < 0.001
  1. *

    Pearson chi-squared test, with < 0.05 indicating significant number of HON-accredited websites returned during a search.

  1. †

    HON-accredited website.

  1. ‡

    Not HON-accredited website.

  1. §

    Percentage of HON-accredited websites, calculated by HONcode+ divided by the total websites (HONcode+) + (HONcode-)).

  1. Sum.

  1. **

    Median.

Fig. 2 Clustered column graph of percentage of HON-accredited websites for keywords arranged by tertiles.

Fig. 2

Clustered column graph of percentage of HON-accredited websites for keywords arranged by tertiles.

Quality assurance

The manual assessment of websites matched the results of our algorithm, confirming its fidelity. For the first 150 “arthritis” (English) results, 20 websites were HON-accredited and 130 were not. We found that 9 (9/150; 6%) of those non-accredited sites could potentially be HON-accredited when assessed manually.

HON accreditation associated variables

Logistic regression analysis demonstrated that there were significant differences between search terms being HON-accredited (Table III). For language, an accredited site was more likely to be found in French than in English, German or Spanish, which were equally likely to return HON-accredited websites. For tertiles, the second tertile was more likely than the third tertile to have HON-accredited sites.

Website sponsor analysis

Sponsorship analysis of the 150 English websites (Table VI) indicated that the most frequent sponsors were government/education (39%), followed by orthopaedic specialists/professional organizations (26%), commercial (18%), other’ (7%), non-profit (6%), and other healthcare professionals (3%). ‘Lawyer’ sponsored far less sites (< 1%).

Table VI.

Website sponsor analysis.

Category/search term Lawyer, n (%) Non-profit, n (%) Government/ education, n (%) Commercial, n (%) Orthopaedic/ professional organizations, n (%) Other health professionals, n (%) Other, n (%) p-value*
Condition
Arthritis 0 (0) 17 (11) 40 (27) 53 (35) 4 (3) 6 (4) 30 (20)
Osteoarthritis 0 (0) 17 (11) 56 (37) 47 (31) 14 (9) 13 (9) 3 (20)
End-stage arthritis 0 (0) 12 (8) 67 (45) 36 (24) 11 (7) 5 (5) 19 (13)
Bone on bone arthritis 1 (1) 17 (11) 41 (27) 42 (28) 18 (8) 11 (7) 19 (13)
Total N/A N/A N/A N/A N/A N/A N/A < 0.001
Arthroplasty
Total joint arthroplasty 0 (0) 10 (7) 85 (57) 18 (12) 20 (13) 2 (1) 15 (10)
Total joint replacement 0 (0) 11 (7) 87 (58) 12 (8) 33 (22) 1 (1) 6 (4)
Total N/A N/A N/A N/A N/A N/A N/A 0.124
Hip
Total hip arthroplasty 0 (0) 7 (5) 80 (53) 19 (13) 31 (21) 3 (2) 10 (7)
Total hip replacement 0 (0) 10 (7) 33 (22) 10 (7) 84 (56) 2 (1) 10 (7)
Total N/A N/A N/A N/A N/A N/A N/A < 0.001
Knee
Total knee arthroplasty 0 (0) 6 (4) 70 (47) 22 (15) 28 (19) 3 (2) 20 (13)
Total knee replacement 0 (0) 7 (5) 34 (23) 17 (11) 77 (51) 7 (5) 8 (5)
Total N/A N/A N/A N/A N/A N/A N/A < 0.001
Approach
Anterior hip replacement 0 (0) 3 (2) 57 (38) 10 (7) 70 (47) 2 (1) 8 (5)
Posterior hip replacement 0 (0) 3 (2) 54 (36) 25 (17) 60 (40) 0 (0) 8 (5)
Total N/A N/A N/A N/A N/A N/A N/A 0.098
Unicompartmentalknee
Half knee replacement 0 (0) 7 (5) 46 (31) 30 (20) 60 (40) 4 (3) 3 (2)
Unicompartmental knee replacement 0 (0) 4 (3) 76 (42) 25 (17) 39 (26) 3 (2) 3 (2)
Mako knee 0 (0) 1 (1) 61 (41) 21(14) 58 (39) 0 (0) 9 (6)
Oxford knee replacement 0 (0) 9 (6) 68 (45) 24 (16) 35 (23) 1 (1) 13 (9)
Total N/A N/A N/A N/A N/A N/A N/A < 0.001
Recovery
Total hip replacement recovery time 0 (0) 10 (7) 57 (38) 33 (22) 39 (26) 6 (4) 5 (3)
Total knee replacement recovery time 0 (0) 13 (9) 53 (35) 38 (25) 30 (20) 12 (8) 4 (3)
Total N/A N/A N/A N/A N/A N/A N/A 0.525
Mean (%) 1 (1) 164 (6) 1,065 (39) 482 (18) 711 (26) 81 (3) 193 (7) < 0.001

  1. N/A, not applicable.

  1. *

    Pearson chi-squared test, where < 0.05 indicates significant difference in number of website sponsors.

Search terms with larger percentages of government/education also had a larger HON-accredited website percentage. These were the ‘condition’ (arthritis, osteoarthritis, end-stage arthritis, bone on bone arthritis), ‘hip’ (total hip arthroplasty, total hip replacement), and ‘knee’ (total knee arthroplasty, total knee replacement) categories (p < 0.001).

Discussion

This study aimed to quantify the quality of TJR-related online information. We found a substantial variation in the quality of websites returned per search term. As determined by our constructed algorithm, the percentage of website HON accreditation was low for all keywords. There were essentially no differences in HON-accredited websites by language. The first tertile contained the majority of HON-accredited websites most frequently. Governmental or educational institutes sponsored almost 50% of websites, while nearly a third of websites were led by orthopaedic specialists or professional organizations. Importantly, nearly a fifth of them were commercially sponsored websites.

Generally, trustworthy resources distributing health information are scarce, and websites with accurate TJR-related information are no exception. Search terms within the categories of condition and arthroplasty only had 16% and 13% of HON-accredited websites, respectively. This is less than websites supporting information for urology or surgical oncology conditions,19,29 but more than others e.g. gynaecological oncology (15%),31 penile cancer (10.4%),35 and benign prostate hyperplasia (9%).30 In the latter study,30 7% of websites were HON-accredited for the category of ‘surgical treatments’, similar to our category of ‘approach’, which also had 7% of HON-accredited websites. This reflects our hypothesis that reliable, accurate TJR and other online health information is lacking.

These results suggest that patients will encounter poor quality information about arthritis conditions and arthroplasty procedures. As both patients and clinicians may struggle to assess website quality, distrust of orthopaedic internet resources may occur. Worse still, poor information may lead to poor decision-making. A number of previous studies demonstrate poor quality website information across different languages.15,16,18 In our study, the percentage of HON-accredited websites were comparable between English (9%) and French, German and Spanish searches (11% to 13%). Similar to thoracic surgery,36 there is less variation of TJR information between languages than results from earlier studies,19 albeit still poor. There is clearly a dearth of reliable online information on TJR, transcending country, and language.

There is a clear propensity for the first 50 websites to contain the majority of HON-accredited websites than the second or third 50. However, despite patients seldom searching further than the first ten results,16 patients may still not find the reliable information they need. Commercial interests may explain this pattern, with websites made more prominent to search engines for a premium cost through marketing techniques like pay-per-click advertising. Furthermore, different search platforms may influence the type of websites and information returned. Depending on whether Google or a similar search engine, such as Bing or DuckDuckGo, is used, websites may appear in different orders, or not at all. A future study could explore if identical websites appear on the first page of different search engines.

In the digital information era, commercial and marketing initiatives are influencing health information exponentially, which may compromise their impartial insights.37 The majority of sponsors in this study comprised government/education or Orthopaedic surgeons/professional organisations. However, almost one in three websites with commercial sponsors suggests that TJR websites may be more influenced by marketing forces, as compared to previously analyzed medical disciplines.19,29-31 Importantly, search terms where these commercial sponsors were more apparent also contained a larger proportion of HON-accredited websites. This may indicate that more objective groups are striving to produce more accurate TJR-related information for the public.

Limitations

Despite HONcode being practical, accessible and validated, several limitations need to be considered. There may be websites with truly reliable information that do not fulfil HONcode criteria, and vice-versa. For example, The Australian Government’s Repatriation Medical Authority or the Australian Clinical Practice Guidelines are not HON-accredited, nor is Scotland’s National Health Service website, nor several leading American university hospitals. These websites appeared numerous times in our search. It is possible these websites are accredited by other tools (Table I). Since 2015, HONcode accreditation is a paid service for which website owners voluntarily apply to have their website HON-accredited. Like previous studies, manual assessment showed that 6% of websites from our control term (“arthritis”) could have HON accreditation.19,29-31 Thus, flaws of HONcode may include voluntary application and relative unawareness from patients and clinicians.

Moreover, search engines like Google also utilize geographical features that only allow local search results to be returned. This may be problematic for non-tech-savvy patients wanting to learn more about their condition if their city or country does not support the most reliable and recent health information. Research has also shown how social media and health-related YouTube videos influence healthcare.38 Patients must heed caution when consuming health information from these largely unregulated media. Hence, concurrently upskilling patient eHealth literacy9 may be crucial for patients searching the internet successfully.

Implications

Health websites facilitate patients’ understanding of their medical issues.39 As such, an opportunity exists to develop and utilize accessible and reliable digital health information tools that support patients when required.40 Clinicians should encourage patients to download quality assessment tools like HONcode or could use these themselves to identify and direct patients to reliable websites. This may enhance patient-clinician rapport,41 informed consent, decision making, and help patients address sensitive health complaints (e.g. urology, gynaecology, and penile cancers)19,20,35,42 In an increasingly digital world with an ageing population, healthcare professionals may serve a critical role in helping to direct patients to the most reliable resources and tools, thereby reducing both patient and their own burden.2

Clinicians should consider the shortage of reliable TJR-related information on the internet across search terms, language and tertiles. Awareness of this poor quality is essential for clinicians to educate and empower patients to conduct thorough health research to obtain superior health literacy. Clinicians can take the initiative to identify and guide patients to reliable and true information on websites.

Correspondence should be sent to Peter F. Choong; E-mail:

References

1. Hesse BW , Greenberg AJ , Rutten LJF . The role of Internet resources in clinical oncology: promises and challenges . Nat Rev Clin Oncol . 2016 ; 13 ( 12 ): 767 - 776 . Crossref PubMed Google Scholar

2. Hungerford DS . Internet access produces misinformed patients: managing the confusion . Orthopedics . 2009 ; 32 ( 9 ): 658 - 660 . Crossref PubMed Google Scholar

3. Risk A , Dzenowagis J . Review of Internet health information quality initiatives . J Med Internet Res . 2001 ; 3 ( 4 ): E28 . Crossref PubMed Google Scholar

4. Couper MP , Singer E , Levin CA , et al. Use of the Internet and ratings of information sources for medical decisions: results from the decisions survey . Med Decis Making . 2010 ; 30 ( 5 Suppl ): 106 - 114 . Crossref PubMed Google Scholar

5. Fox S , Pew DM . Internet and American life project . 2020 . https://www.pewresearch.org/internet/2013/01/15/health-online-2013/ (date last accessed 05 March 2020 ). Google Scholar

6. Nagler RH , Gray SW , Romantan A , et al. Differences in information seeking among breast, prostate, and colorectal cancer patients: results from a population-based survey . Patient Educ Couns . 2010 ; 81 ( suppl ): S54 - S62 . Crossref PubMed Google Scholar

7. Berland GK , Elliott MN , Morales LS , et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish . JAMA . 2001 ; 285 ( 20 ): 2612 - 2621 . Crossref PubMed Google Scholar

8. Bruce JG , Tucholka JL , Steffens NM , Neuman HB . Quality of online information to support patient decision-making in breast cancer surgery . J Surg Oncol . 2015 ; 112 ( 6 ): 575 - 580 . Crossref PubMed Google Scholar

9. Norman CD , Skinner HA . eHealth literacy: essential skills for consumer health in a networked world . J Med Internet Res . 2006 ; 8 ( 2 ): e9 . Crossref PubMed Google Scholar

10. Norman CD , Skinner HA . eHEALS: the eHealth literacy scale . J Med Internet Res . 2006 ; 8 ( 4 ): e27 . Google Scholar

11. Del Giudice P , Bravo G , Poletto M , et al. Correlation between eHealth literacy and health literacy using the eHealth literacy scale and real-life experiences in the health sector as a proxy measure of functional health literacy: cross-sectional web-based survey . J Med Internet Res . 2018 ; 20 ( 10 ): e281 . Crossref PubMed Google Scholar

12. Arcury TA , Sandberg JC , Melius KP , et al. Older adult Internet use and eHealth literacy . Journal of applied gerontology: the official journal of the Southern Gerontological Society . 2018 ; 733464818807468 . Crossref PubMed Google Scholar

13. Mueller J , Jay C , Harper S , et al. Web use for symptom appraisal of physical health conditions: a systematic review . J Med Internet Res . 2017 ; 19 ( 6 ): e202 . Crossref PubMed Google Scholar

14. Chen X , Siu LL . Impact of the media and the Internet on oncology: survey of cancer patients and oncologists in Canada . J Clin Oncol . 2001 ; 19 ( 23 ): 4291 - 4297 . Crossref PubMed Google Scholar

15. Foundation HOtN . The HON code of conduct for medical and health web sites. HON Foundation Ginebra . 2009 . https://www.hon.ch/HONcode/ (date last accessed 05 March 2020 ). Google Scholar

16. Eysenbach G , Köhler C . How do consumers search for and appraise health information on the world wide web? qualitative study using focus groups, usability tests, and in-depth interviews . BMJ . 2002 ; 324 ( 7337 ): 573 - 577 . Crossref PubMed Google Scholar

17. Gaudinat A , Grabar N , Boyer C . Machine learning approach for automatic quality criteria detection of health web Pages . Stud Health Technol Inform . 2007 ; 129 ( Pt 1 ): 705 - 709 . PubMed Google Scholar

18. Eastham JA . Robotic-Assisted prostatectomy: is there truth in advertising? Eur Urol . 2008 ; 54 ( 4 ): 720 - 722 . Crossref PubMed Google Scholar

19. Lawrentschuk N , Abouassaly R , Hackett N , Groll R , Fleshner NE . Health information quality on the Internet in urological oncology: a multilingual longitudinal evaluation . Urology . 2009 ; 74 ( 5 ): 1058 - 1063 . Crossref PubMed Google Scholar

20. Graves S , Davidson D , de Steiger R , Tomkins A . Australian orthopaedic association national joint replacement registry annual report . Australian Orthopaedic Association National Joint Replacement Registry . 2014 . Google Scholar

21. Ekman A , Hall P , Litton J-E . Can we trust cancer information on the Internet?--A comparison of interactive cancer risk sites . Cancer Causes Control . 2005 ; 16 ( 6 ): 765 - 772 . Crossref PubMed Google Scholar

22. Charnock D , Shepperd S . Learning to discern online: applying an appraisal tool to health websites in a workshop setting . Health Educ Res . 2004 ; 19 ( 4 ): 440 - 446 . Crossref PubMed Google Scholar

23. Dobbins M , Watson S , Read K , et al. A tool that assesses the evidence, transparency, and usability of online health information: development and reliability assessment . JMIR Aging . 2018 ; 1 ( 1 ): e3 . Crossref PubMed Google Scholar

24. Ltd M . The LIDA instrument . UK : Minervation Ltd Oxford , 2007 . http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf Google Scholar

25. Devine T , Broderick J , Harris LM , Wu H , Hilfiker SW . Making quality health websites a national public health priority: toward quality Standards . J Med Internet Res . 2016 ; 18 ( 8 ): e211 . Crossref PubMed Google Scholar

26. Avery KNL , Blazeby JM , Lane JA , et al. Decision-Making about PSA testing and prostate biopsies: a qualitative study embedded in a primary care randomised trial . Eur Urol . 2008 ; 53 ( 6 ): 1186 - 1193 . Crossref PubMed Google Scholar

27. Smith RP , Devine P , Jones H , et al. Internet use by patients with prostate cancer undergoing radiotherapy . Urology . 2003 ; 62 ( 2 ): 273 - 277 . Crossref PubMed Google Scholar

28. Boyer C , Baujard V , Griesser V , Scherrer JR . HONselect: a multilingual and intelligent search tool integrating heterogeneous web resources . Int J Med Inform . 2001 ; 64 ( 2-3 ): 253 - 258 . Crossref PubMed Google Scholar

29. Lawrentschuk N , Sasges D , Tasevski R , et al. Oncology health information quality on the Internet: a multilingual evaluation . Ann Surg Oncol . 2012 ; 19 ( 3 ): 706 - 713 . Crossref PubMed Google Scholar

30. Chen EC , Manecksha RP , Abouassaly R , et al. A multilingual evaluation of current health information on the Internet for the treatments of benign prostatic hyperplasia . Prostate Int . 2014 ; 2 ( 4 ): 161 - 168 . Crossref PubMed Google Scholar

31. Hewitt E , Mulcahy A , Lawrentschuk N , et al. Health information quality on the Internet in gynaecological oncology: a multilingual evaluation . International journal of gynecological cancer . 2014 . Lippincott Williams & Wilkins . Google Scholar

32. Gargalionis J . Honcoder: a Python3 interface to the HONcode website certification database [Github] . 2019 . https://github.com/johngarg/honcoder (date last accessed 05 March 2020 ). Google Scholar

33. Killeen S , Hennessey A , El Hassan Y , et al. Gastric cancer-related information on the Internet: incomplete, poorly accessible, and overly commercial . Am J Surg . 2011 ; 201 ( 2 ): 171 - 178 . Crossref PubMed Google Scholar

34. Alkhateeb S , Lawrentschuk N . Consumerism and its impact on robotic-assisted radical prostatectomy . BJU Int . 2011 ; 108 ( 11 ): 1874 - 1878 . Crossref PubMed Google Scholar

35. Teh J , Op't Hoog S , Nzenza T , et al. Penile cancer information on the Internet: a needle in a haystack . BJU Int . 2018 ; 122 Suppl 5 ( suppl 5 ): 22 - 26 . Crossref PubMed Google Scholar

36. Davaris M , Barnett S , Abouassaly R , Lawrentschuk N . Thoracic surgery information on the Internet: a multilingual quality assessment . Interact J Med Res . 2017 ; 6 ( 1 ): e5 . Crossref PubMed Google Scholar

37. Mayer MA , Karkaletsis V , Stamatakis K . MedIEQ-quality labelling of medical web content using multilingual . Medical and Care Compunetics 3 . 2006 : 121 - 183 . PubMed Google Scholar

38. Gabarron E , Fernandez-Luque L , Armayones M , Lau AY . Identifying measures used for assessing quality of YouTube videos with patient health information: a review of current literature . Interact J Med Res . 2013 ; 2 ( 1 ): e6 . Crossref PubMed Google Scholar

39. Hoppe IC . Readability of patient information regarding breast cancer prevention from the web site of the National cancer Institute . J Cancer Educ . 2010 ; 25 ( 4 ): 490 - 492 . Crossref PubMed Google Scholar

40. Cassidy JT , Baker JF . Orthopaedic patient information on the world wide web: an essential review . J Bone Joint Surg Am . 2016 ; 98 ( 4 ): 325 - 338 . Crossref PubMed Google Scholar

41. Sechrest RC . The Internet and the physician-patient relationship . Clin Orthop Relat Res . 2010 ; 468 ( 10 ): 2566 - 2571 . Crossref PubMed Google Scholar

42. Lawrentschuk N , Sasges D , Tasevski R , et al. Oncology health information quality on the Internet: a multilingual evaluation . Ann Surg Oncol . 2012 ; 19 ( 3 ): 706 - 713 . Crossref PubMed Google Scholar

Author contributions

M. T. Davaris: Set up the project, Collected and analyzed data, Wrote and edited the manuscript.

M. M. Dowsey: Set up the project, Analyzed the data, Wrote and edited the manuscript.

S. Bunzli: Set up the project, Wrote and edited the manuscript.

P. F. Choong: Set up the project, Wrote and edited the manuscript.

Funding statement

No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article.

ICMJE COI statement

P. Choong declares personal fees paid by Stryker Orthopaedics. Depuy Johnson & Johnson, Ziimer, and Kluwer, and grants paid by Medacta and Depuy. M. Dowsey declares grants paid by Medacta, National Health & Medical Research Council, Australian Research Council, and Pfizer.

Acknowledgements

M. Davaris is supported through the National Health and Medical Research Council (NHMRC) Centre of Research Excellence in Total Joint Replacement (APP1116325) at the Department of Surgery, St Vincent’s Hospital. M. Dowsey holds a NHMRC Career Development Fellowship (APP1122526) and University of Melbourne Dame Kate Campbell Fellowship. P. Choong holds a NHMRC Practitioner Fellowship (APP1154203). We thank John Gargalionis for creating the algorithm to collect HONcode data and A/Prof Nathan Lawrentschuk for his previous work that inspired this study.

Ethical review statement

Quality assurance approval (092/19) was obtained at St Vincent’s Hospital, Melbourne.

Twitter

Follow M. T. Davaris @DavarisMyles

Follow M. M. Dowsey @OPUS_TJR

© 2020 Author(s) et al. This is an open-access article distributed under the terms of the Creative Commons Attributions licence (CC-BY-NC-ND), which permits unrestricted use, distribution, and reproduction in any medium, but not for commercial gain, provided the original author and source are credited.

Information

Journal

Bone & Joint Open

Volume

1 No.4 | Pages 64 - 73

Section

Arthroplasty

Published

20 April 2020

DOI

Expand all

PhD Candidate, Medical Doctor

Department of Surgery, St Vincent’s Hospital, University of Melbourne, Melbourne, Australia

Search for more articles by this author

Associate Professor of Research

Department of Surgery, St Vincent’s Hospital, University of Melbourne, Melbourne, Australia

Search for more articles by this author

Physiotherapist, Research Associate

Department of Surgery, St Vincent’s Hospital, University of Melbourne, Melbourne, Australia

Search for more articles by this author

Department of Surgery, St Vincent’s Hospital, University of Melbourne, Melbourne, Australia

pchoong@unimelb.edu.au

Search for more articles by this author
Orthopaedic
Surgery
Arthroplasty
Internet
Patient education
Websites
Submit Letter to the Editor

Share

Share article on social media
Facebook
X (Twitter)
LinkedIn
Email

Figures

Fig. 1 Column graph of median percentage of HON-accredited sites for all keywords arranged according to language. Each keyword was searched on native Google search engine of respective countries.

Fig. 1

Column graph of median percentage of HON-accredited sites for all keywords arranged according to language. Each keyword was searched on native Google search engine of respective countries.
Go to Article
Fig. 2 Clustered column graph of percentage of HON-accredited websites for keywords arranged by tertiles.

Fig. 2

Clustered column graph of percentage of HON-accredited websites for keywords arranged by tertiles.
Go to Article

Metrics

Downloaded 742 times

References

1. Find in article

Hesse BW , Greenberg AJ , Rutten LJF . The role of Internet resources in clinical oncology: promises and challenges . Nat Rev Clin Oncol . 2016 ; 13 ( 12 ): 767 - 776 .

Crossref PubMed Google Scholar
2. Find in article

Hungerford DS . Internet access produces misinformed patients: managing the confusion . Orthopedics . 2009 ; 32 ( 9 ): 658 - 660 .

Crossref PubMed Google Scholar
3. Find in article

Risk A , Dzenowagis J . Review of Internet health information quality initiatives . J Med Internet Res . 2001 ; 3 ( 4 ): E28 .

Crossref PubMed Google Scholar
4. Find in article

Couper MP , Singer E , Levin CA , et al. Use of the Internet and ratings of information sources for medical decisions: results from the decisions survey . Med Decis Making . 2010 ; 30 ( 5 Suppl ): 106 - 114 .

Crossref PubMed Google Scholar
5. Find in article

Fox S , Pew DM . Internet and American life project . 2020 . (date last accessed 05 March 2020 ).

https://www.pewresearch.org/internet/2013/01/15/health-online-2013/ Google Scholar
6. Find in article

Nagler RH , Gray SW , Romantan A , et al. Differences in information seeking among breast, prostate, and colorectal cancer patients: results from a population-based survey . Patient Educ Couns . 2010 ; 81 ( suppl ): S54 - S62 .

Crossref PubMed Google Scholar
7. Find in article

Berland GK , Elliott MN , Morales LS , et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish . JAMA . 2001 ; 285 ( 20 ): 2612 - 2621 .

Crossref PubMed Google Scholar
8. Find in article

Bruce JG , Tucholka JL , Steffens NM , Neuman HB . Quality of online information to support patient decision-making in breast cancer surgery . J Surg Oncol . 2015 ; 112 ( 6 ): 575 - 580 .

Crossref PubMed Google Scholar
9. Find in article

Norman CD , Skinner HA . eHealth literacy: essential skills for consumer health in a networked world . J Med Internet Res . 2006 ; 8 ( 2 ): e9 .

Crossref PubMed Google Scholar
10. Find in article

Norman CD , Skinner HA . eHEALS: the eHealth literacy scale . J Med Internet Res . 2006 ; 8 ( 4 ): e27 .

Google Scholar
11. Find in article

Del Giudice P , Bravo G , Poletto M , et al. Correlation between eHealth literacy and health literacy using the eHealth literacy scale and real-life experiences in the health sector as a proxy measure of functional health literacy: cross-sectional web-based survey . J Med Internet Res . 2018 ; 20 ( 10 ): e281 .

Crossref PubMed Google Scholar
12. Find in article

Arcury TA , Sandberg JC , Melius KP , et al. Older adult Internet use and eHealth literacy . Journal of applied gerontology: the official journal of the Southern Gerontological Society . 2018 ; 733464818807468 .

Crossref PubMed Google Scholar
13. Find in article

Mueller J , Jay C , Harper S , et al. Web use for symptom appraisal of physical health conditions: a systematic review . J Med Internet Res . 2017 ; 19 ( 6 ): e202 .

Crossref PubMed Google Scholar
14. Find in article

Chen X , Siu LL . Impact of the media and the Internet on oncology: survey of cancer patients and oncologists in Canada . J Clin Oncol . 2001 ; 19 ( 23 ): 4291 - 4297 .

Crossref PubMed Google Scholar
15. Find in article

Foundation HOtN . The HON code of conduct for medical and health web sites. HON Foundation Ginebra . 2009 . (date last accessed 05 March 2020 ).

https://www.hon.ch/HONcode/ Google Scholar
16. Find in article

Eysenbach G , Köhler C . How do consumers search for and appraise health information on the world wide web? qualitative study using focus groups, usability tests, and in-depth interviews . BMJ . 2002 ; 324 ( 7337 ): 573 - 577 .

Crossref PubMed Google Scholar
17. Find in article

Gaudinat A , Grabar N , Boyer C . Machine learning approach for automatic quality criteria detection of health web Pages . Stud Health Technol Inform . 2007 ; 129 ( Pt 1 ): 705 - 709 .

PubMed Google Scholar
18. Find in article

Eastham JA . Robotic-Assisted prostatectomy: is there truth in advertising? Eur Urol . 2008 ; 54 ( 4 ): 720 - 722 .

Crossref PubMed Google Scholar
19. Find in article

Lawrentschuk N , Abouassaly R , Hackett N , Groll R , Fleshner NE . Health information quality on the Internet in urological oncology: a multilingual longitudinal evaluation . Urology . 2009 ; 74 ( 5 ): 1058 - 1063 .

Crossref PubMed Google Scholar
20. Find in article

Graves S , Davidson D , de Steiger R , Tomkins A . Australian orthopaedic association national joint replacement registry annual report . Australian Orthopaedic Association National Joint Replacement Registry . 2014 .

Google Scholar
21. Find in article

Ekman A , Hall P , Litton J-E . Can we trust cancer information on the Internet?--A comparison of interactive cancer risk sites . Cancer Causes Control . 2005 ; 16 ( 6 ): 765 - 772 .

Crossref PubMed Google Scholar
22. Find in article

Charnock D , Shepperd S . Learning to discern online: applying an appraisal tool to health websites in a workshop setting . Health Educ Res . 2004 ; 19 ( 4 ): 440 - 446 .

Crossref PubMed Google Scholar
23. Find in article

Dobbins M , Watson S , Read K , et al. A tool that assesses the evidence, transparency, and usability of online health information: development and reliability assessment . JMIR Aging . 2018 ; 1 ( 1 ): e3 .

Crossref PubMed Google Scholar
25. Find in article

Devine T , Broderick J , Harris LM , Wu H , Hilfiker SW . Making quality health websites a national public health priority: toward quality Standards . J Med Internet Res . 2016 ; 18 ( 8 ): e211 .

Crossref PubMed Google Scholar
26. Find in article

Avery KNL , Blazeby JM , Lane JA , et al. Decision-Making about PSA testing and prostate biopsies: a qualitative study embedded in a primary care randomised trial . Eur Urol . 2008 ; 53 ( 6 ): 1186 - 1193 .

Crossref PubMed Google Scholar
27. Find in article

Smith RP , Devine P , Jones H , et al. Internet use by patients with prostate cancer undergoing radiotherapy . Urology . 2003 ; 62 ( 2 ): 273 - 277 .

Crossref PubMed Google Scholar
28. Find in article

Boyer C , Baujard V , Griesser V , Scherrer JR . HONselect: a multilingual and intelligent search tool integrating heterogeneous web resources . Int J Med Inform . 2001 ; 64 ( 2-3 ): 253 - 258 .

Crossref PubMed Google Scholar
29. Find in article

Lawrentschuk N , Sasges D , Tasevski R , et al. Oncology health information quality on the Internet: a multilingual evaluation . Ann Surg Oncol . 2012 ; 19 ( 3 ): 706 - 713 .

Crossref PubMed Google Scholar
30. Find in article

Chen EC , Manecksha RP , Abouassaly R , et al. A multilingual evaluation of current health information on the Internet for the treatments of benign prostatic hyperplasia . Prostate Int . 2014 ; 2 ( 4 ): 161 - 168 .

Crossref PubMed Google Scholar
31. Find in article

Hewitt E , Mulcahy A , Lawrentschuk N , et al. Health information quality on the Internet in gynaecological oncology: a multilingual evaluation . International journal of gynecological cancer . 2014 . Lippincott Williams & Wilkins .

Google Scholar
32. Find in article

Gargalionis J . Honcoder: a Python3 interface to the HONcode website certification database [Github] . 2019 . (date last accessed 05 March 2020 ).

https://github.com/johngarg/honcoder Google Scholar
33. Find in article

Killeen S , Hennessey A , El Hassan Y , et al. Gastric cancer-related information on the Internet: incomplete, poorly accessible, and overly commercial . Am J Surg . 2011 ; 201 ( 2 ): 171 - 178 .

Crossref PubMed Google Scholar
34. Find in article

Alkhateeb S , Lawrentschuk N . Consumerism and its impact on robotic-assisted radical prostatectomy . BJU Int . 2011 ; 108 ( 11 ): 1874 - 1878 .

Crossref PubMed Google Scholar
35. Find in article

Teh J , Op't Hoog S , Nzenza T , et al. Penile cancer information on the Internet: a needle in a haystack . BJU Int . 2018 ; 122 Suppl 5 ( suppl 5 ): 22 - 26 .

Crossref PubMed Google Scholar
36. Find in article

Davaris M , Barnett S , Abouassaly R , Lawrentschuk N . Thoracic surgery information on the Internet: a multilingual quality assessment . Interact J Med Res . 2017 ; 6 ( 1 ): e5 .

Crossref PubMed Google Scholar
37. Find in article

Mayer MA , Karkaletsis V , Stamatakis K . MedIEQ-quality labelling of medical web content using multilingual . Medical and Care Compunetics 3 . 2006 : 121 - 183 .

PubMed Google Scholar
38. Find in article

Gabarron E , Fernandez-Luque L , Armayones M , Lau AY . Identifying measures used for assessing quality of YouTube videos with patient health information: a review of current literature . Interact J Med Res . 2013 ; 2 ( 1 ): e6 .

Crossref PubMed Google Scholar
39. Find in article

Hoppe IC . Readability of patient information regarding breast cancer prevention from the web site of the National cancer Institute . J Cancer Educ . 2010 ; 25 ( 4 ): 490 - 492 .

Crossref PubMed Google Scholar
40. Find in article

Cassidy JT , Baker JF . Orthopaedic patient information on the world wide web: an essential review . J Bone Joint Surg Am . 2016 ; 98 ( 4 ): 325 - 338 .

Crossref PubMed Google Scholar
41. Find in article

Sechrest RC . The Internet and the physician-patient relationship . Clin Orthop Relat Res . 2010 ; 468 ( 10 ): 2566 - 2571 .

Crossref PubMed Google Scholar
42. Find in article

Lawrentschuk N , Sasges D , Tasevski R , et al. Oncology health information quality on the Internet: a multilingual evaluation . Ann Surg Oncol . 2012 ; 19 ( 3 ): 706 - 713 .

Crossref PubMed Google Scholar