Introduction: Idiopathic pulmonary fibrosis (IPF) results in lung function decline. Prognostic models that accurately predict IPF progression could inform research studies and clinical care. Objectives: To develop deep learning (DL) models to predict IPF progression using baseline high-resolution computed tomography (HRCT). Methods: Retrospective analysis was performed on IPF patients enrolled in clinical trials (NCT01872689, NCT00287729, NCT01366209). Only baseline visit HRCT (non-contrast, supine position, full inspiration) were included in the analysis. The image dataset was split into training (n = 274) and holdout (n = 117). The training dataset was then split into 5 folds for cross-validation (CV). Two multi-task DL models [HRCT-only and multi-modal (HRCT and baseline clinical features)] were trained to simultaneously predict 3 endpoints: FVC at 1 year (mL), FVC change at 1 year (mL) and FVC slope (mL/year). The performance of the DL models were benchmarked with a linear model using baseline clinical features and evaluated using squared Pearson correlation coefficient (r²). Results: The multi-modal model had the best CV performance on training set with mean r² of 0.87, 0.13, and 0.14 for FVC at 1 year, FVC change at 1 year, and FVC slope. On the holdout set, the same model showed r² of 0.88, 0.11, and 0.12. In comparison, the benchmark model had a mean r² of 0.85, 0.05, and 0.05 on the training set and 0.89, 0.04, and 0.04 on the holdout set, respectively, for the 3 endpoints. Conclusion: HRCT scans add marginal value to baseline clinical features in predicting IPF progression. Further work is required to improve the performance of the current models for potential use in research studies and clinical care.